Tag Archives: Python

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 36: Select Active Camera in OpenCV With Voice Commands

Welcome back, makers, engineers, and AI enthusiasts! In our previous lessons, we built out robust multi-camera setups, streaming feeds from both Raspberry Pi cameras and high-definition USB cameras. But watching four feeds in separate tiles is only half the battle. What happens when you want to interact with your system hands-free?

In this lesson, we are taking our edge vision projects to the next level by integrating voice commands. You will learn how to run a dedicated speech-to-text thread in the background, catch voice triggers safely using a thread-safe queue, and dynamically switch your active main camera feed in OpenCV on the fly—just by speaking.

What You Will Learn in This Lesson

  • Multi-Threaded Speech Recognition: How to run the stt listener in a separate daemon thread so it never blocks or stutters your high-FPS video processing loop.

  • Thread-Safe Communication: Using Python’s Queue module to pass voice commands seamlessly from the background listening thread into your main application loop.

  • Handling STT Variations: Accounting for common speech-to-text homophone variations (like “two”, “to”, and “too”, or “four” and “for”) to make your voice control robust and reliable.

  • Dynamic Frame Routing: Mapping spoken commands to specific camera streams (piCam1, piCam2, usbCam1, and usbCam2) and updating your primary display window instantly.

  • Organized Window Layouts: Positioning and sizing multiple OpenCV windows on your desktop workspace for a clean, professional multi-camera dashboard.

Step-by-Step Breakdown of the Script

1. Setting Up the Background Voice Thread

When working with real-time video processing in OpenCV, blocking functions are your worst enemy. If you call a speech recognition listening function directly inside your main while loop, your video frames will freeze while waiting for audio input.

To prevent this, we initialize a background worker function (getCamera) and launch it as a daemon thread:

  • The thread continuously listens for your voice commands using the Fusion Hat STT module.

  • Once a command is captured, it strips whitespace, checks for exit triggers (like saying 'quit'), and pushes valid commands directly into our commandQ.

2. Initializing Multiple Cameras

Our setup harnesses the full power of the hardware by combining native Raspberry Pi camera interfaces with standard USB webcams:

  • Pi Cameras: Configured via Picamera2 at a crisp 1280x720 resolution with RGB888 formatting running smoothly at 60 frames per second.

  • USB Cameras: Initialized through OpenCV’s VideoCapture class, set to target resolution and optimized for 30 FPS.

3. Processing and Routing Commands in the Main Loop

Inside the primary application loop, our script handles three core tasks simultaneously:

  1. Calculate Performance: Continuously tracks and smooths out the frames-per-second (FPS) metric so you can monitor system load.

  2. Check the Command Queue: Non-blockingly checks if the background thread has dropped a new camera selection into commandQ. If a new command is waiting, it updates the mainCam variable.

  3. Route the Main Frame: Evaluates the active camera string—including clever fallback checks for common voice misinterpretations like 'camera to' or 'camera for'—and assigns the corresponding video stream to mainFrame.

4. Managing Your OpenCV Windows

To give you a complete command-center experience, the script generates a large primary display window for your active camera view, accompanied by a clean row of smaller preview tiles across the bottom of your screen for all four connected feeds.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 35: Running Multiple Pi Cameras and USB Cameras on the Pi 5

AI on the Edge LESSON 35: Running Multiple Pi Cameras and USB Cameras on the Pi 5

Hello everybody! Paul McWhorter here from toptechboy.com, welcoming you back to another thrilling session of our AI on the Edge series. Today, we are taking our hardware vision capabilities to the next level. If you have ever wondered how to scale up your Raspberry Pi 5 setup beyond just a single lens, this lesson is for you. We are going to write a Python script that pulls live feeds from multiple Raspberry Pi cameras and multiple USB webcams simultaneously, displaying all four streams cleanly in real time with an on-screen frames-per-second (FPS) tracker.

Codex Knowledge: Multi-Camera Architecture on the Pi 5

When working with edge hardware like the Raspberry Pi 5, managing multiple high-bandwidth video streams requires understanding how the underlying libraries interact with the Linux kernel and system memory. In this lesson, we leverage two distinct hardware interfaces:

  • Picamera2 API: We initialize two separate native Pi camera instances using Picamera2(0) and Picamera2(1). By explicitly configuring the preview size to 640×360, setting the format to RGB888, locking the frame rate to 60 FPS, and calling align() before starting, we ensure the hardware pipelines are optimized for low-latency streaming without frame-drop bottlenecks.
  • OpenCV VideoCapture: For our USB webcams, we use OpenCV’s cv2.VideoCapture() mapped to specific device indices (in our setup, indices 16 and 18). We explicitly set the frame width, height, and target frame rate properties to keep the data flow synchronized with our Pi camera streams.
  • Window Layout Management: Using OpenCV highgui functions like cv2.namedWindow, cv2.moveWindow, and cv2.resizeWindow, we programmatically arrange all four camera feeds into a neat 2×2 grid on your desktop workspace, preventing windows from stacking blindly on top of each other.

General Knowledge: The Evolution of Multi-Stream Machine Vision

In industrial automation, robotics, and advanced edge AI deployments, relying on a single camera angle is rarely enough. Multi-camera systems are the gold standard for comprehensive spatial awareness, 3D depth estimation, object tracking across wide fields of view, and panoramic monitoring. Historically, running multiple high-resolution video streams required bulky, power-hungry desktop workstations equipped with expensive capture cards. Today, single-board computers like the Raspberry Pi 5—combined with optimized kernel drivers and efficient software wrappers like Picamera2—allow engineers and creators to build robust, multi-sensor vision arrays right at the edge at a fraction of the cost and power consumption.

Python Source Code

Here is the complete, production-ready script for Lesson 35. Make sure your cameras are securely connected and properly indexed before running the program.

Conclusion

There you have it! You are now successfully driving a multi-camera computer vision array right off your Raspberry Pi 5. Play around with the window positioning, check your device indices if your USB cameras don’t immediately pop up, and get ready because in our next lesson we will start piping these multi-source frames directly into our neural network inference models. Keep tinkering, stay curious, and I will see you in the next lesson!

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 34 SUPPLEMENT: Simple Improvement to FPS

In this video I show you how to dramatically improve the FPS of our work in lesson 34. I also show the solution to the issue of the Fusion Hat microphone not working with the project.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 34: Project Combining TTS, STT, Face Recognition and Servos on Pi 5

In our previous lessons, we built individual components. We got our machine to listen locally using Speech-to-Text (STT), we got it to talk back cleanly with Text-to-Speech (TTS), and we learned how to manipulate physical hardware using precision servos. Today, we bridge the gap between software and the physical world. We are building a multi-threaded, autonomous edge system that tracks humanoids in real time.

The Engineering Mindset: Concurrency & Threading

If you try to build a system like this sequentially—running camera capture, object detection, audio listening, and audio speaking in a single while True: loop—your project will fail completely. Why? Because tasks like waiting for a voice command (stt.listen()) or synthesizing a voice line are blocking operations. If your code is stuck waiting for you to finish speaking a command, your camera frames freeze, your servo adjustments drop to zero, and your humanoid target escapes.

To solve this, we architect our software using Multi-threading. We spawn independent execution paths that run concurrently, passing data back and forth safely using thread-safe Queues:

  • The Main Thread: Handles the high-speed Picamera2 video capture loop, executes Haar Cascade face/eye calculations, and updates the servo angles via mathematical error tracking.
  • The Speak Thread: Idles quietly in the background until a message lands in the speakQ, instantly triggering the local Piper TTS engine without stalling our camera frame rate.
  • The Command Thread: Keeps a continuous ear open via our local STT engine. When it parses a voice command like “track”, “release”, or “quit”, it safely pops that token into the commandQ for the main loop to execute on its next pass.

The Complete Fusion Architecture Code

Below is the complete Python pipeline developed in today’s lesson. Make sure your hardware connections for the pan-and-tilt servos match pins 2 and 3 on your expansion setup, and verify your fusion_hat software stack is completely updated.

Breaking Down the Math & Logic

Pro-Tip on State Debouncing: Notice how we handle messages using the cnt variable and the msgOld buffer. If we simply asked the system to say “Humanoid Detected” every single time the frame loops, the TTS engine would crash from queue overflow, stuttering continuously. By checking if msgOld != msg, we ensure the system speaks a notification exactly once upon transition. The 25-frame hysteresis buffer ensures that a single missed frame doesn’t cause a false “No Humanoid” trigger.

Proportional Error Tracking

Look carefully at how the servo angles are adjusted:

We are calculating the error vector—how many pixels the center of the bounding box (xBoxC) is from the absolute center of our camera stream (xFrameC). Instead of moving the servo by a fixed step, we move it proportionally to the size of the error. Big error? Fast movement. Small error? Gentle, microscopic correction. Dividing by 50 dampens our proportional gain loop so our servos don’t violently oscillate and throw our camera system completely out of alignment.

Your Homework Assignment

You have the system reading frames, listening asynchronously, and adjusting angles proportionally. Now it’s time to earn your stripes as an edge systems designer.

Your Assignment: Look at our msg tracking logic. Right now, if a humanoid is detected, it simply asks, “Shall I track?”. Your task is to tie our speech input directly to this state. Modify the control flow so that the tracking logic requires verification. If a humanoid is detected, the system must wait until the commandQ yields a verified verbal confirmation—like “execute” or “yes”—before the proportional servo tracking begins. If it hears “abort”, it must break off and look away.

Drop your custom implementation script in the comments section below, explain the logic behind your state modifications, and let’s see how you optimize your loop efficiency. Go get ’em!

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

Welcome, Makers!

Well, hello there! It is absolutely fantastic to have you back. I’m Paul McWhorter, and today, we are taking a massive step forward in our AI on the Edge journey.

Up until now, we’ve been hard-coding our color thresholds (those pesky Lower Color and Upper Color values) to tell our camera what to look for. That’s fine for a science experiment, but it’s not exactly “smart,” is it? If the lighting changes, or if we want to track a different colored object, we have to go back into the code and manually edit those numbers.

Not anymore!

In today’s lesson, we are building a tool that lets us teach the AI. We’re going to use the mouse to draw a Region of Interest (ROI) right on our camera feed. The system will look at the pixels inside that box, calculate the average Hue, Saturation, and Value, and automatically set our tracking range for us.

This is the kind of professional-level functionality that turns a hobby project into a true, intelligent machine.

The Concept: From Hard-Coding to Dynamic Learning

The magic happens in our mouseAction function. Instead of just reading pixel values, we are now implementing a “click-and-drag” system:

  1. Click and Hold: We capture the startX and startY coordinates.

  2. Drag: We draw a rectangle in real-time so we can see exactly what area we are selecting.

  3. Release: We take that specific slice of the image, convert it to the HSV color space, and use the cv2.mean() function to find the average color properties.

  4. Auto-Tune: We set our LC (Lower Color) and UC (Upper Color) based on that average.

By doing this, the system learns what “object” we want to track on the fly. It’s elegant, it’s powerful, and it feels like real magic when you see those servos snap onto your target after a quick mouse drag.

What We’ve Accomplished

By the end of this lesson, you will have a system that:

  • Visually selects an object using the mouse.

  • Automatically calculates the optimal HSV thresholds for that specific object.

  • Updates the tracking behavior immediately without needing to stop or re-run the code.

  • Maintains that professional “Edge” feel, giving you real-time feedback on your FPS and mouse position data.

A Note on the “Edge”

Remember, we aren’t just running code; we are running on hardware. When we calculate the mean of the ROI, we are doing real image processing on the fly. You’ll notice the Composite and Mask windows updated immediately, giving you a visual confirmation that your “teacher” (you!) has successfully guided the “student” (the AI).

This is the power of working with OpenCV and the Raspberry Pi. You are building a system that observes, thinks, and reacts—all in real-time.

Get Ready to Build

Grab your Pi, make sure your servos are ready to go, and let’s get that camera calibrated. You’ve put in the work to get this far, and today is where all that effort starts to feel really rewarding.

I’m incredibly proud of how far you’ve come. Let’s dive in and start building!

Are you ready to see how accurately your Pi can “see” once you’ve given it the ability to learn from your selections?

In the video lesson we developed the following code.