Category Archives: AI On the Edge

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 35: Running Multiple Pi Cameras and USB Cameras on the Pi 5

Hello everybody! Paul McWhorter here from toptechboy.com, welcoming you back to another thrilling session of our AI on the Edge series. Today, we are taking our hardware vision capabilities to the next level. If you have ever wondered how to scale up your Raspberry Pi 5 setup beyond just a single lens, this lesson is for you. We are going to write a Python script that pulls live feeds from multiple Raspberry Pi cameras and multiple USB webcams simultaneously, displaying all four streams cleanly in real time with an on-screen frames-per-second (FPS) tracker.

Codex Knowledge: Multi-Camera Architecture on the Pi 5

When working with edge hardware like the Raspberry Pi 5, managing multiple high-bandwidth video streams requires understanding how the underlying libraries interact with the Linux kernel and system memory. In this lesson, we leverage two distinct hardware interfaces:

  • Picamera2 API: We initialize two separate native Pi camera instances using Picamera2(0) and Picamera2(1). By explicitly configuring the preview size to 640×360, setting the format to RGB888, locking the frame rate to 60 FPS, and calling align() before starting, we ensure the hardware pipelines are optimized for low-latency streaming without frame-drop bottlenecks.
  • OpenCV VideoCapture: For our USB webcams, we use OpenCV’s cv2.VideoCapture() mapped to specific device indices (in our setup, indices 16 and 18). We explicitly set the frame width, height, and target frame rate properties to keep the data flow synchronized with our Pi camera streams.
  • Window Layout Management: Using OpenCV highgui functions like cv2.namedWindow, cv2.moveWindow, and cv2.resizeWindow, we programmatically arrange all four camera feeds into a neat 2×2 grid on your desktop workspace, preventing windows from stacking blindly on top of each other.

General Knowledge: The Evolution of Multi-Stream Machine Vision

In industrial automation, robotics, and advanced edge AI deployments, relying on a single camera angle is rarely enough. Multi-camera systems are the gold standard for comprehensive spatial awareness, 3D depth estimation, object tracking across wide fields of view, and panoramic monitoring. Historically, running multiple high-resolution video streams required bulky, power-hungry desktop workstations equipped with expensive capture cards. Today, single-board computers like the Raspberry Pi 5—combined with optimized kernel drivers and efficient software wrappers like Picamera2—allow engineers and creators to build robust, multi-sensor vision arrays right at the edge at a fraction of the cost and power consumption.

Python Source Code

Here is the complete, production-ready script for Lesson 35. Make sure your cameras are securely connected and properly indexed before running the program.

Conclusion

There you have it! You are now successfully driving a multi-camera computer vision array right off your Raspberry Pi 5. Play around with the window positioning, check your device indices if your USB cameras don’t immediately pop up, and get ready because in our next lesson we will start piping these multi-source frames directly into our neural network inference models. Keep tinkering, stay curious, and I will see you in the next lesson!

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 34 SUPPLEMENT: Simple Improvement to FPS

In this video I show you how to dramatically improve the FPS of our work in lesson 34. I also show the solution to the issue of the Fusion Hat microphone not working with the project.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 34: Project Combining TTS, STT, Face Recognition and Servos on Pi 5

In our previous lessons, we built individual components. We got our machine to listen locally using Speech-to-Text (STT), we got it to talk back cleanly with Text-to-Speech (TTS), and we learned how to manipulate physical hardware using precision servos. Today, we bridge the gap between software and the physical world. We are building a multi-threaded, autonomous edge system that tracks humanoids in real time.

The Engineering Mindset: Concurrency & Threading

If you try to build a system like this sequentially—running camera capture, object detection, audio listening, and audio speaking in a single while True: loop—your project will fail completely. Why? Because tasks like waiting for a voice command (stt.listen()) or synthesizing a voice line are blocking operations. If your code is stuck waiting for you to finish speaking a command, your camera frames freeze, your servo adjustments drop to zero, and your humanoid target escapes.

To solve this, we architect our software using Multi-threading. We spawn independent execution paths that run concurrently, passing data back and forth safely using thread-safe Queues:

  • The Main Thread: Handles the high-speed Picamera2 video capture loop, executes Haar Cascade face/eye calculations, and updates the servo angles via mathematical error tracking.
  • The Speak Thread: Idles quietly in the background until a message lands in the speakQ, instantly triggering the local Piper TTS engine without stalling our camera frame rate.
  • The Command Thread: Keeps a continuous ear open via our local STT engine. When it parses a voice command like “track”, “release”, or “quit”, it safely pops that token into the commandQ for the main loop to execute on its next pass.

The Complete Fusion Architecture Code

Below is the complete Python pipeline developed in today’s lesson. Make sure your hardware connections for the pan-and-tilt servos match pins 2 and 3 on your expansion setup, and verify your fusion_hat software stack is completely updated.

Breaking Down the Math & Logic

Pro-Tip on State Debouncing: Notice how we handle messages using the cnt variable and the msgOld buffer. If we simply asked the system to say “Humanoid Detected” every single time the frame loops, the TTS engine would crash from queue overflow, stuttering continuously. By checking if msgOld != msg, we ensure the system speaks a notification exactly once upon transition. The 25-frame hysteresis buffer ensures that a single missed frame doesn’t cause a false “No Humanoid” trigger.

Proportional Error Tracking

Look carefully at how the servo angles are adjusted:

We are calculating the error vector—how many pixels the center of the bounding box (xBoxC) is from the absolute center of our camera stream (xFrameC). Instead of moving the servo by a fixed step, we move it proportionally to the size of the error. Big error? Fast movement. Small error? Gentle, microscopic correction. Dividing by 50 dampens our proportional gain loop so our servos don’t violently oscillate and throw our camera system completely out of alignment.

Your Homework Assignment

You have the system reading frames, listening asynchronously, and adjusting angles proportionally. Now it’s time to earn your stripes as an edge systems designer.

Your Assignment: Look at our msg tracking logic. Right now, if a humanoid is detected, it simply asks, “Shall I track?”. Your task is to tie our speech input directly to this state. Modify the control flow so that the tracking logic requires verification. If a humanoid is detected, the system must wait until the commandQ yields a verified verbal confirmation—like “execute” or “yes”—before the proportional servo tracking begins. If it hears “abort”, it must break off and look away.

Drop your custom implementation script in the comments section below, explain the logic behind your state modifications, and let’s see how you optimize your loop efficiency. Go get ’em!

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 33: Tracking Faces with Pan Tilt Camera in OpenCV on Pi 5

Hey guys, Paul McWhorter here from toptechboy.com. In today’s lesson, we are taking our “AI on the Edge” skills to the next level. We aren’t just detecting faces anymore; we are going to make our camera system react to them.

We are integrating our computer vision logic with physical hardware to create a pan-tilt tracking system. We’ll use the Raspberry Pi 5 to run high-speed inference, detect a face, calculate exactly how far that face has drifted from the center of our frame, and then command our servos to follow it in real-time. It is one thing to see a box draw around a face; it is a completely different level of “cool” when the camera actually turns to look at you.

The Engineering Concept: The Error Loop

In robotics, this is a classic control problem. We have a Target (the center of the face) and a Setpoint (the center of our camera frame). The difference between these two points is our Error.

  • xError: How far left or right the face is from the center.
  • yError: How far up or down the face is from the center.

By taking that error and dividing it by a “gain” constant (in our case, 50/2), we can smoothly adjust the servo angles. If we don’t divide by a constant, the camera will snap aggressively to the target or overshoot it. This simple division creates a “proportional” response that keeps our tracking smooth and precise.

What to Focus On

Make sure you have your picamera2 and fusion_hat libraries updated and configured before you dive in. The key to this lesson isn’t just getting the servos to move—it’s understanding how to bridge the gap between the coordinates returned by OpenCV and the angle coordinates required by your servo library.

Pay close attention to how we calculate the center of the frame and the center of the detected face box. Once you understand that math, you can use this same logic to track anything: faces, colored objects, or even specific shapes!

Homework Assignment: Show Your Work!

Alright guys, no excuses! If you want to truly master this hardware, you cannot just sit there and watch me do it—you have to write the code and run it yourself.

Your Task: Get this tracking system working. Once you have it tracking a face, I want you to experiment with that “Gain” factor (the 50/2 part). Try increasing it and decreasing it. What happens to the tracking quality? Is it smoother? Does it jitter?

Record a video of your camera following your face around the room, upload it to YouTube, and link back to this video at the top of your page. Post a link to your homework in the comments section below so I can see you running your code like a boss.

 

The code we developed in the video is available below.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 32: Facial Recognition and Eye Tracking in OpenCV

Hey guys, Paul McWhorter here from TopTechBoy.com. Welcome back to our AI on the Edge series. If you’ve been following along, you already know how to pull high-frame-rate video off your Raspberry Pi 5 using the new picamera2 library, and you know how to use OpenCV to hunt down faces in a crowded frame.

But today, we are taking things a massive step forward. We aren’t just looking for faces anymore—we are looking inside the face to track the eyes.

This lesson highlights one of the most vital concepts in all of computer vision: The Region of Interest (ROI). If you try to scan an entire 1280×720 frame for tiny features like eyes, your frame rate will absolutely tank. Instead, we are going to act like real engineers. We will use a cascading logic approach: find the face first, isolate that exact box, and search only inside that small window for the eyes.

Go ahead and pour yourself a nice, cold glass of iced coffee or a hot cup of black coffee, get your code ready, and let’s break down exactly how this program works.

This is the code we developed in the video:

Code Architecture & Codex Breakdown

Since you already have the script loaded up in your IDE, let’s dissect the critical logic gates that make this tracking script fast and accurate.

1. Setting Up the High-Performance Pipeline

We configure the Picamera2 frontend to grab a crisp RGB888 array at a resolution of 1280×720 targeting 60 FPS. By using .capture_array(), we bypass slow formatting overhead and feed raw pixel data directly into OpenCV. Because the camera orientation might be flipped depending on your desktop mounting rig, we use cv2.flip(frame, -1) to keep the spatial coordinates intuitive.

2. The Cascading Filter Matrix

Notice how we initialize two distinct classifiers using pre-trained Haar Cascades:

  • haarcascade_frontalface_default.xml (To grab the macro features of the face)

  • haarcascade_eye.xml (To grab the micro features of the eyes)

We pass a minSize parameter of 100×100 pixels for the face detector. Why? Because we don’t care about background noise or tiny false positives across the room. We want to find you, sitting right in front of the workstation.

3. The Magic of the Region of Interest (ROI)

This is where the real engineering happens. Look closely at this inner loop:

Instead of passing the massive gray frame to the eye finder, we slice the array: gray[y:y+h, x:x+w]. This isolates a tiny sub-matrix containing nothing but your face. The search area drops exponentially, keeping our frame rates close to maximum velocity.

4. Re-Mapping Local to Global Coordinates

When the eye detector finds a match inside the sliced face frame, it returns local coordinates (i, j, w, h) relative to the top-left corner of that face box, not the whole screen. If you tried to draw a rectangle directly at (i, j), your eye boxes would be floating erratically in the top-left corner of your monitor!

To fix this spatial offset, we map them back to global coordinate space by adding the face’s original offsets:

  • Global X Position: x + i

  • Global Y Position: y + j

General Knowledge: How Haar Cascades and ROIs Work Under the Hood

Now that you understand the mechanics of the script, let’s dive into the fundamental computer vision theory that makes legacy Edge AI tracking so efficient.

The Viola-Jones Framework

Haar Cascade classifiers are based on the Viola-Jones object detection framework. Instead of using massive, compute-heavy deep learning neural networks that require powerful discrete GPUs, Haar Cascades utilize simple, binary pixel-intensity features called Haar-like features.

These features act like digital templates looking for specific shifts in brightness:

  • Edge Features: Detects boundaries where a dark zone transitions into a light zone (like the bridge of your nose versus your cheek).

  • Line Features: Useful for identifying long, horizontal elements like eyebrows or the line of the mouth.

  • Center-Surround Features: Excellent for finding eyes, where the dark pupil is surrounded by lighter skin and sclera.

Why Slicing the Array Saves Your Processor

Every time you invoke .detectMultiScale(), OpenCV has to pass a sliding window across the image matrix at multiple scales, performing thousands of additions and subtractions per frame.

Mathematically, if an entire frame has a pixel area, scanning it scales linearly with that total area. By filtering for the face first and establishing a tight Region of Interest (ROI), you reduce the eye tracking search space down to a fractional area.

On resource-constrained hardware like an edge microcontroller or a single-board computer, isolating the matrix dimensions before calling nested lookups is the difference between a sluggish, unusable slideshow and a silky smooth tracking experience.