Tag Archives: Fusion HAT+

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 40: Active Face Tracker with Pan Tilt Camera and MediaPipe on Pi 5

Boys and girls, welcome back! In today’s lesson, we are going to tie together everything we’ve been building in the AI on the Edge series and construct something truly interactive: a fully autonomous, voice-controlled, pan-tilt face tracking robot running locally right on your Raspberry Pi 5!

In our previous lessons, we learned how to detect faces using MediaPipe and how to drive physical servos to point a camera. Today, we step up our game. We are bringing in multithreading, Speech-to-Text (STT) using the Fusion Hat, and Text-to-Speech (TTS) with Piper to give our Pi a voice, a personality, and the physical ability to track down humanoids in real time.

What We Are Building in This Lesson

Imagine setting up a camera system that constantly scans its environment. The moment a human face enters the frame, the system locks on and speaks up: “Humanoid Detected, Shall I track?”

Using real-time voice commands, you can issue directions straight to the Pi without touching a keyboard:

  • “Track” — Activates proportional control on the pan-tilt kit. The servos will calculate pixel error relative to the center of the frame and smoothly adjust their angles to keep your face dead center.

  • “Release” — Disables active tracking, letting the servos hold their position while the vision loop continues monitoring.

  • “Blind” — Isolates the facial keypoints for the subject’s eyes and draws solid black circles over them in real time, causing the robot to announce: “Subject Has Been Blinded, Shall I Vaporize?”

  • “Restore” — Removes the eye overlay and brings vision back to normal.

  • “Quit” — Safely terminates all background threads, announces shutdown, and closes down the application gracefully.

Key Technical Concepts Covered

1. Multi-Threaded Architecture & Thread-Safe Queues

Audio processing—both listening for voice input and generating spoken speech—is computationally heavy and blocking by nature. If you run speech recognition directly inside your primary video processing loop, your frame rate will plummet from a smooth 60 FPS down to a complete crawl.

To solve this, we spin up two independent background threads using Python’s threading module:

  • Speech Thread: Monitors a thread-safe speakQ (Queue) and handles text-to-speech output using Piper without stalling the main loop.

  • Command Thread: Continuously listens to the microphone via Speech-to-Text, strips and parses incoming voice triggers, and pushes valid commands into a commandQ.

2. MediaPipe Facial Landmark Detection

We leverage MediaPipe’s high-speed face detection solution running at 1280×720 resolution on the Raspberry Pi 5. By calculating relative bounding boxes and keypoint coordinate matrices (x, y), the system identifies both face centroids and precise feature locations like eye coordinates.

3. Proportional Servo Error Correction

To keep the camera centered on a moving subject, the script computes positional error delta values between the center of the bounding box and the exact midpoint of the camera frame:

xError = xBoxCenter – xFrameCenter

yError = yBoxCenter – yFrameCenter

These error values are scaled down and applied directly to update the current pan and tilt servo angles, ensuring smooth, continuous tracking movement without jarring overshoots.

Your Homework Assignment

Get your Raspberry Pi 5, mount your pan-tilt camera assembly with the Fusion Hat, and implement the multithreaded architecture outlined in this lesson. Tune your servo scaling factors to ensure your tracking motion is fluid and responsive at 60 FPS. Have fun!


 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 36: Select Active Camera in OpenCV With Voice Commands

Welcome back, makers, engineers, and AI enthusiasts! In our previous lessons, we built out robust multi-camera setups, streaming feeds from both Raspberry Pi cameras and high-definition USB cameras. But watching four feeds in separate tiles is only half the battle. What happens when you want to interact with your system hands-free?

In this lesson, we are taking our edge vision projects to the next level by integrating voice commands. You will learn how to run a dedicated speech-to-text thread in the background, catch voice triggers safely using a thread-safe queue, and dynamically switch your active main camera feed in OpenCV on the fly—just by speaking.

What You Will Learn in This Lesson

  • Multi-Threaded Speech Recognition: How to run the stt listener in a separate daemon thread so it never blocks or stutters your high-FPS video processing loop.

  • Thread-Safe Communication: Using Python’s Queue module to pass voice commands seamlessly from the background listening thread into your main application loop.

  • Handling STT Variations: Accounting for common speech-to-text homophone variations (like “two”, “to”, and “too”, or “four” and “for”) to make your voice control robust and reliable.

  • Dynamic Frame Routing: Mapping spoken commands to specific camera streams (piCam1, piCam2, usbCam1, and usbCam2) and updating your primary display window instantly.

  • Organized Window Layouts: Positioning and sizing multiple OpenCV windows on your desktop workspace for a clean, professional multi-camera dashboard.

Step-by-Step Breakdown of the Script

1. Setting Up the Background Voice Thread

When working with real-time video processing in OpenCV, blocking functions are your worst enemy. If you call a speech recognition listening function directly inside your main while loop, your video frames will freeze while waiting for audio input.

To prevent this, we initialize a background worker function (getCamera) and launch it as a daemon thread:

  • The thread continuously listens for your voice commands using the Fusion Hat STT module.

  • Once a command is captured, it strips whitespace, checks for exit triggers (like saying 'quit'), and pushes valid commands directly into our commandQ.

2. Initializing Multiple Cameras

Our setup harnesses the full power of the hardware by combining native Raspberry Pi camera interfaces with standard USB webcams:

  • Pi Cameras: Configured via Picamera2 at a crisp 1280x720 resolution with RGB888 formatting running smoothly at 60 frames per second.

  • USB Cameras: Initialized through OpenCV’s VideoCapture class, set to target resolution and optimized for 30 FPS.

3. Processing and Routing Commands in the Main Loop

Inside the primary application loop, our script handles three core tasks simultaneously:

  1. Calculate Performance: Continuously tracks and smooths out the frames-per-second (FPS) metric so you can monitor system load.

  2. Check the Command Queue: Non-blockingly checks if the background thread has dropped a new camera selection into commandQ. If a new command is waiting, it updates the mainCam variable.

  3. Route the Main Frame: Evaluates the active camera string—including clever fallback checks for common voice misinterpretations like 'camera to' or 'camera for'—and assigns the corresponding video stream to mainFrame.

4. Managing Your OpenCV Windows

To give you a complete command-center experience, the script generates a large primary display window for your active camera view, accompanied by a clean row of smaller preview tiles across the bottom of your screen for all four connected feeds.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 35: Running Multiple Pi Cameras and USB Cameras on the Pi 5

Hello everybody! Paul McWhorter here from toptechboy.com, welcoming you back to another thrilling session of our AI on the Edge series. Today, we are taking our hardware vision capabilities to the next level. If you have ever wondered how to scale up your Raspberry Pi 5 setup beyond just a single lens, this lesson is for you. We are going to write a Python script that pulls live feeds from multiple Raspberry Pi cameras and multiple USB webcams simultaneously, displaying all four streams cleanly in real time with an on-screen frames-per-second (FPS) tracker.

Codex Knowledge: Multi-Camera Architecture on the Pi 5

When working with edge hardware like the Raspberry Pi 5, managing multiple high-bandwidth video streams requires understanding how the underlying libraries interact with the Linux kernel and system memory. In this lesson, we leverage two distinct hardware interfaces:

  • Picamera2 API: We initialize two separate native Pi camera instances using Picamera2(0) and Picamera2(1). By explicitly configuring the preview size to 640×360, setting the format to RGB888, locking the frame rate to 60 FPS, and calling align() before starting, we ensure the hardware pipelines are optimized for low-latency streaming without frame-drop bottlenecks.
  • OpenCV VideoCapture: For our USB webcams, we use OpenCV’s cv2.VideoCapture() mapped to specific device indices (in our setup, indices 16 and 18). We explicitly set the frame width, height, and target frame rate properties to keep the data flow synchronized with our Pi camera streams.
  • Window Layout Management: Using OpenCV highgui functions like cv2.namedWindow, cv2.moveWindow, and cv2.resizeWindow, we programmatically arrange all four camera feeds into a neat 2×2 grid on your desktop workspace, preventing windows from stacking blindly on top of each other.

General Knowledge: The Evolution of Multi-Stream Machine Vision

In industrial automation, robotics, and advanced edge AI deployments, relying on a single camera angle is rarely enough. Multi-camera systems are the gold standard for comprehensive spatial awareness, 3D depth estimation, object tracking across wide fields of view, and panoramic monitoring. Historically, running multiple high-resolution video streams required bulky, power-hungry desktop workstations equipped with expensive capture cards. Today, single-board computers like the Raspberry Pi 5—combined with optimized kernel drivers and efficient software wrappers like Picamera2—allow engineers and creators to build robust, multi-sensor vision arrays right at the edge at a fraction of the cost and power consumption.

Python Source Code

Here is the complete, production-ready script for Lesson 35. Make sure your cameras are securely connected and properly indexed before running the program.

Conclusion

There you have it! You are now successfully driving a multi-camera computer vision array right off your Raspberry Pi 5. Play around with the window positioning, check your device indices if your USB cameras don’t immediately pop up, and get ready because in our next lesson we will start piping these multi-source frames directly into our neural network inference models. Keep tinkering, stay curious, and I will see you in the next lesson!

AI On the Edge, Tutorial

AI on the Edge LESSON 23: Creating Regions of Interest (ROI) in OpenCV with Slicing

Welcome back, everyone! In this lesson, we are stepping into a foundational aspect of computer vision: manipulation of specific regions within a video frame.

Up to this point, we have been grabbing the full frame from our camera and performing operations on the entire image. But in real-world edge AI and robotics applications, processing every single pixel of a high-resolution frame is an absolute waste of compute power. If you want to detect a license plate, track a face, or monitor a specific sensor layout on a machine, you don’t need to look at the sky or the floor. You need to isolate a Region of Interest (ROI).

In this lesson, you will learn how to use Python’s powerful matrix slicing capabilities to chop up a frame, isolate specific quadrants, manipulate pixels inside an ROI, and display multiple synchronized windows across your desktop without crashing your system footprint.

The Core Concept: Image Slicing and ROIs

In OpenCV, an image frame isn’t just a visual picture—it is a standard NumPy array. A color frame is a 3D matrix structured by rows, columns, and color channels: [Rows, Columns, Channels] or [Height, Width, Color].

Because it is a standard array, we can use standard Python slicing notation to isolate any rectangular box we want:

ROI = frame[rowstart : rowend,  colstart : colend]

The .copy() Trap

When you slice a piece of an array in Python like ROI = frame[0:100, 0:100], Python does not create a new image in your RAM. It creates a view or a pointer back to the original frame. If you modify pixels inside that ROI, you will accidentally alter your original main camera frame!

To isolate a region and modify it independently without bleeding back into your primary frame, you must explicitly use the .copy() method:

Below is the complete code script we built during the video tutorial. Copy this code exactly into your Python environment, verify your geometry setups, and run it.

Homework Assignment

Alright, it is time to earn your stripes and see if you can fly with the big dogs. Your homework assignment is to take this foundation and build a dynamic tracking target box using the array geometry principles we just learned.

  1. Create a single main camera window (640 x 360).

  2. Draw an independent rectangular ROI box that starts directly in the dead center of the screen.

  3. Using your keyboard parameters (cv2.waitKey), program the system so that using the Arrow Keys (or ‘i’, ‘j’, ‘k’, ‘l’) smoothly updates variables to move the ROI box dynamically around the screen in real-time.

  4. Crucial Constraint: Do not let your boundary indices drift off the array! You must write conditional boundaries so that if your moving target hits the edge of your $640 \times 360$ boundary layout, it locks at the frame border and prevents an out-of-bounds index crash.

  5. In a separate output window, display only the contents of the moving target box in real-time grayscaled format.

Grab your morning coffee, fire up your code editor, write the script from scratch, and do not copy-paste code you don’t understand. Leave a link to your homework solution video in the YouTube comments section so I can see your progress!

AI On the Edge

AI on the Edge LESSON 22: Understanding Pictures and Video Frames as a Data Structure

Hey guys, Paul McWhorter here with TopTechBoy.com, and today we are diving into the heart of computer vision. We’ve been playing around with getting images from the camera, but have you ever stopped to actually look at what a picture is when it’s inside your computer’s memory?

If you want to be a master of AI on the Edge, you have to stop thinking about images as “pictures” and start seeing them as what they really are: a massive, organized grid of numbers.

What is a Picture, Really?

In this lesson, we are peeling back the curtain on how OpenCV and Python handle video frames. When we call piCam.capture_array(), we aren’t just taking a snapshot; we are pulling a data array into memory.

Think of it like a giant spreadsheet where every single cell is a pixel.

  • Dimensions: Your image has a width and a height, which correspond to the number of rows and columns in that array. It is important to remember the row designator comes first, then the column, [ R, C]

  • The Depth (The RGB Channels): It’s not just a flat 2D grid! Each “cell” in that grid is actually a little sub-array containing three values: Red, Green, and Blue. That is why we call it a 3D data structure.

Manipulating Data, Not Just Pixels

The magic happens when you realize you can reach into that array and change those numbers directly.

In the code we developed today, we aren’t just displaying video; we are performing data science on video frames. We explored how to:

  1. Access individual pixels: By referencing specific coordinates in our frame array, we can pull out the color data for a single spot.

  2. Draw shapes by modifying arrays: Notice how we don’t need a “draw square” function to put a box on the screen? We simply tell a slice of that array to equal [0, 0, 255]. We are literally changing the color values of those pixels to solid red.

  3. Regions of Interest (ROI): This is critical for AI. You don’t always need to look at the whole frame. We learned how to “slice” the array to isolate a Region of Interest. By carving out a smaller piece of that memory, we can perform operations—like converting to grayscale—on just that section, which saves a massive amount of processing power.

Why Does This Matter?

If you want to build a robot that recognizes objects or tracks faces, you need to understand this structure. AI models don’t “see” a cat; they see a mathematical representation of that cat’s pixel values. By learning how to slice, manipulate, and convert these arrays, you are learning the fundamental language of machine learning.

We are building the foundation here, folks. Once you get comfortable with how to manipulate these arrays, we are going to start doing some really cool stuff with image processing and filtering.

Dive into that code, change those array values, and see what happens when you mess with the dimensions! Don’t just run it—experiment with it.

I’ll see you guys in the next lesson!

In this lesson we developed the following code: