Tag Archives: Raspberry Pi 5

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 58: Object Detection With Speech to Text Voice Control on Raspberry Pi 5

Welcome back, my friends! This is Paul McWhorter from toptechboy.com, and welcome to AI on the Edge LESSON 58: Object Detection With Speech to Text Voice Control on Raspberry Pi 5!

In this lesson, we take a massive leap forward in edge artificial intelligence and robotics by giving our Raspberry Pi 5 camera rig the gift of speech. Up until now, changing our target object of interest required editing our Python script and restarting the code. In Lesson 58, we bridge the gap between computer vision and natural human interaction by introducing local Speech-to-Text (STT) and Piper Text-to-Speech (TTS). Now, you can simply speak to your system, tell it what object you want to track, and watch as your pan-tilt camera locks on and follows your command!

Understanding the Program Architecture

The program is structured around a robust, multi-threaded architecture designed to handle audio recognition and high-speed computer vision simultaneously without lagging your video stream.

1. Voice-Driven Input via Background Threading

To ensure our camera stream maintains a smooth frame rate, we initialize a dedicated background worker thread using Python’s threading library. This thread runs independently, listening to microphone input via local STT tools.

  • When you speak your object of interest, the thread captures the audio, strips and lowercases the text, and checks for termination commands like quitting the program.

  • If a valid target is spoken, the thread pushes the string into a thread-safe communication queue and commands Piper TTS to talk back, audibly confirming your choice with a phrase like “Now Tracking”.

2. Thread-Safe Command Queuing

Because audio processing and computer vision operate on different timing loops, we use Python’s thread-safe queue mechanism. The background listening thread drops new target requests into the queue, while our main loop checks on every frame pass whether a new object of interest has arrived. This prevents race conditions and keeps our variable updates clean and reliable.

3. High-Speed OpenCV and MediaPipe Vision Loop

The main execution thread runs our primary while loop, capturing high-resolution frames from the Picamera2 camera module at 60 FPS, flipping the orientation, and feeding the frame data into MediaPipe’s efficient object detection model.

  • When detections match our active target class, the program extracts bounding box coordinates, calculates the geometric center of the object, renders visual confirmation boxes and confidence percentages on the screen, and computes pixel error offsets from the center of the display.

4. Proportional Servo Feedback Control

Once the pixel error between the target center and the screen center is calculated, our servo update function kicks in. By applying proportional gain multipliers to the horizontal and vertical error values, the script smoothly drives the SunFounder Fusion HAT servos, keeping your target locked right in the crosshairs.

Hardware Requirements

  • Raspberry Pi 5

  • Raspberry Pi Camera Module (configured with Picamera2 at 1280 by 720 resolution and 60 FPS)

  • SunFounder Fusion HAT (for driving pan and tilt servos)

  • USB Microphone (for local speech-to-text input)

  • Speaker or Headphones (for Piper text-to-speech audio feedback)

Homework Assignment for Next Week

Now that you have full voice-controlled object selection up and running, it is time for your next big challenge. For next week’s lesson, take the code provided at the bottom of this page and add mode control with three distinct operating states:

  1. find mode: In this mode, the servos remain completely switched off. The camera stays stationary, but MediaPipe actively finds, labels, and boxes your object of interest whenever it appears in the frame.

  2. track mode: In this mode, the servos are switched on, and the camera actively drives the pan-tilt rig to track and center the object of interest.

  3. scan mode: In this mode, if your target is out of view, the system actively sweeps and scans the room using the servos to find the object, and then automatically switches over to track mode once the object is located.

Grab your coffee, fire up your Raspberry Pi 5, review the complete code at the bottom of this page, and I will see you in the next lesson!

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 57: Detecting and Tracking Objects with MediaPipe

Welcome back, my friends! This is Paul McWhorter from toptechboy.com, and welcome to LESSON 57 of our AI on the Edge tutorial series.

In this lesson, we take what we’ve learned about local object detection using MediaPipe and put it to work in the physical world! We are building an active target tracking system using a 2-DOF pan-tilt servo rig driven by the Raspberry Pi and the Fusion HAT.

By calculating the positional error between the center of our targeted object (in this case, a bottle) and the center of our frame, we compute feedback adjustments to continuously drive our pan and tilt servos—keeping the target locked right in the center of the camera view.

What You Will Learn in Lesson 57

  1. Setting Up MediaPipe Object Detection: Configuring the efficientdet_lite0.tflite model with threshold options and maximum result limits.

  2. Integrating Picamera2 with OpenCV: Capturing high-resolution camera feeds (1280 by 720 at 60 FPS) and converting raw frames to MediaPipe-compatible image formats.

  3. Bounding Box Analysis & Centroid Calculation: Extracting bounding box dimensions and computing the precise pixel center of detected targets.

  4. Proportional Error Servo Control: Using gain coefficients to convert pixel displacements into physical angle corrections for pan and tilt servos.

  5. Deadband Implementation: Preventing servo chatter and hunting by defining a deadband region around the center of the screen.

Hardware Requirements

  • Raspberry Pi 5 (or Raspberry Pi 4)

  • Raspberry Pi Camera Module (configured with Picamera2)

  • SunFounder Fusion HAT (or compatible servo controller HAT)

  • 2-DOF Pan-Tilt Servo Kit (Pan attached to Pin 2, Tilt attached to Pin 3)

  • Target Object (e.g., a water bottle or any object recognized by COCO dataset categories)

How the Proportional Tracking Control Works

To keep the target centered in the video frame, we continuously measure the distance in pixels between the frame’s center and the center of the target’s bounding box:

errorX = boxCenterX – screenCenterX

errorY = boxCenterY – screenCenterY

If the magnitude of the error exceeds our deadBand limit (15 pixels), we adjust the pan and tilt angles proportionally:

panAngle = panAngle – (errorX * panGain)

tiltAngle = tiltAngle + (errorY * tiltGain)

  • Pan Limits: Clamped between -90 and +90 degrees.

  • Tilt Limits: Clamped between -45 and +45 degrees.

Homework Assignment

Now that you have basic single-target tracking running smoothly:

  1. Multi-Target Priority: Modify the script so that if multiple target objects appear on screen, the camera locks onto the object closest to the center of the frame rather than defaulting to the first detected bounding box.

  2. Dynamic Smoothness Filter: Add an exponential moving average or low-pass filter to the bounding box center coordinates to smooth out tracking jitter when the object moves rapidly across the field of view.

  3. Audibly speak the Object of Interst to track. This will require you to incorporate Speech to Text voice recognition.

Get your coffee, fire up your Raspberry Pi, copy the code above into your IDE, and let’s get tracking!

General Artificial Intelligence & Edge Computing Concepts

What is Edge AI?

Edge AI refers to the deployment of artificial intelligence algorithms directly on hardware devices, such as microcontrollers, single-board computers, or edge accelerators, located close to the source of data generation rather than relying on cloud-based processing servers.

Key Advantages of Local Inference:

  • Low Latency: Processing video feeds directly on an edge device eliminates network transmission delay, enabling real-time feedback loops critical for robotics and physical pan-tilt tracking.

  • Privacy & Security: Sensitive visual or audio data remains on the local hardware without requiring continuous streaming to third-party cloud infrastructure.

  • Network Independence: Edge deployments operate reliably in remote or bandwidth-constrained environments without requiring active internet connectivity.

  • Cost Efficiency: Running localized models eliminates recurring per-API-call cloud infrastructure and compute costs.

Proportional-Integral-Derivative Control Concepts:

  • Proportional Control: Calculates an output value proportional to the current error magnitude. High proportional gain results in aggressive responses, while low gain yields sluggish tracking.

  • Deadband: A non-responsive range around the zero-error position where control adjustments are intentionally suspended to prevent motor oscillation, noise amplification, and mechanical wear.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 56: Identifying Object of Interest in Target Zone with Mediapipe

Welcome back, everyone! Paul McWhorter here from toptechboy.com, and welcome to LESSON 56 of our AI on the Edge tutorial series! Grab yourself a nice hot cup of iced coffee (no sugar needed!), sit back, relax, and let’s jump right into today’s project.

AI on the Edge LESSON 56: Identifying Object of Interest in Target Zone with MediaPipe

In our previous lessons, we learned how to use MediaPipe’s object detection task to scan our live camera feed and pick out objects using pre-trained edge AI models like efficientdet_lite0.tflite. We got our bounding boxes drawn, displayed class labels, and showed confidence scores—all running smoothly right on our Raspberry Pi!

Today, we are taking our computer vision setup to the next level by building a Target Zone Detector.

What We Are Doing Today

Instead of just blindly detecting everything in the frame, we want our edge device to actively monitor a specific region of interest—a “Target Zone”—and trigger an alert when a specific Target Class enters that boundary.

Here is what we accomplish in this lesson:

  1. Define a Fixed Target Zone: We calculate a customizable bounding box right in the center of our 1280×720 video feed (e.g., 150×250 pixels).

  2. Filter by Object Class: We set a target object (in this example, a ‘bottle’). The system ignores all other object classes detected in the scene.

  3. Calculate the Bounding Box Center: For the target object, we compute its exact center point (X_center, Y_center) on the fly and draw a tracking dot on it.

  4. Boundary & Collision Logic: We test whether the center coordinates of our target fall inside our designated target zone using the condition: zoneX1 <= X_center <= zoneX2 and zoneY1 <= Y_center <= zoneY2.

  5. Dynamic Visual Feedback:

    • Idle State: When no target is inside the zone, the box remains Green.

    • Active State: The moment the target’s center point steps into the target zone, the boundary box turns Red, and an alert banner—”Target Has Entered the Zone”—is dynamically rendered on the camera feed.

  6. Real-Time Performance: We track frame rate (FPS) using an exponential moving average calculation to keep an eye on edge processing speed.

Homework / Challenge Assignment

Now that you have seen how to set up logic for a single target object in a single zone, here is your homework assignment to solidify what you’ve learned!

  1. We want to now actively track the selected Object of Interest. Turn your servos on on the pan tilt camera, and keep the Object of Interest in the center of the camera view.

Give it a try on your own before watching the solution in the next video. Remember: You don’t learn by watching; you learn by doing!

Discussion & Community

Did your target zone detection work smoothly on your Raspberry Pi setup? How many FPS are you getting with your Pi Camera? Leave a comment below on YouTube to share your results, questions, or modifications!

If you enjoyed this tutorial, hit that Like button, Subscribe to the channel, and share it with anyone who wants to learn real-world AI on the Edge.

As always, I’ll see you guys in the next lesson!

General Knowledge / Context

  • MediaPipe Tasks API: MediaPipe Python Tasks API provides high-level Python interfaces for object detection, landmasking, and classification tasks utilizing lightweight edge models such as TensorFlow Lite (.tflite).

  • PiCamera2 & OpenCV: In modern Raspberry Pi OS (Bullseye/Bookworm), Picamera2 handles native camera hardware capture in raw formats like RGB888, which can then be converted to MediaPipe Image objects (mp.Image) and rendered on OpenCV (cv2) GUI windows seamlessly.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5

Welcome back, everyone! In today’s lesson, AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5, we are building a custom, real-time hand gesture recognition system from scratch. Up until now, we have been using MediaPipe to track keypoints, identify individual hand joints, and extract raw 3D coordinate data. But raw coordinates alone don’t tell the computer what your hand is actually doing. Today, we bridge that gap by teaching our Raspberry Pi 5 how to train on custom hand positions and dynamically classify live hand gestures using pure mathematics and vector geometry—no heavy neural network retraining required.

The core secret to making gesture recognition work reliably across different camera distances is scale-invariant distance normalization. If you simply measure the pixel distance between your fingertips and your wrist, bringing your hand closer to the camera lens will blow out those numbers and break your classifier. To solve this, our script establishes a baseline unit of measurement for every single frame: the distance between the wrist (landmark 0) and the index finger base joint (landmark 5). By dividing all measured finger-tip-to-wrist and tip-to-tip distances by this dynamic scale factor, the resulting feature vector remains identical whether your hand is two feet away or five feet away from the lens.

Once we extract this normalized 15-element feature vector—representing the spatial ratios between all five fingertips relative to the wrist and each other—we construct an interactive training phase directly inside the live video loop. The program prompts you in the terminal for the number of custom gestures you wish to record, along with their labels (such as “Peace”, “Thumbs Up”, or “Fist”). As you present each gesture to your camera feed and press the spacebar, the algorithm captures the instantaneous geometric signature of your hand and stores it in a runtime training dictionary.

During live inference, our classifier calculates the Sum of Absolute Errors (SAE), or Manhattan distance, between the live incoming feature vector and every stored profile in our training dataset. The function loops through all saved gestures to find the candidate with the absolute lowest cumulative error score. To prevent false positives when an untrained or messy hand pose is shown, we compare that lowest error score against a strict threshold value (maxErrorThreshold = 3.5). If the error falls within the allowable boundary, the matched gesture label is instantly displayed on the live OpenCV HUD; otherwise, the system safely defaults to “Unknown”.

We manage our camera feed using high-speed frame capture with Picamera2 running at 60 frames per second at 1280×720 resolution, processing hand detection with MediaPipe Hands, and rendering real-time performance diagnostics—including an exponential moving average FPS counter—directly onto the video output. Go ahead, study the methodology, load the concept onto your Raspberry Pi 5, grab a hot cup of coffee, and let’s get building!

 

AI On the Edge, Raspberry Pi

Hand Recognition on the Raspberry Pi 5 With openCV and Mediapipe

This is a base program which allows you to use mediapipe on the Raspberry Pi 5 under openCV to identify each landmark on a hand. This program is important as we can see the index of each landmark. This will serve as the basis for our next program which will allow us to begin to do gesture recogntion.