Tag Archives: Fusion HAT+

AI on the Edge LESSON 58: Object Detection With Speech to Text Voice Control on Raspberry Pi 5

Welcome back, my friends! This is Paul McWhorter from toptechboy.com, and welcome to AI on the Edge LESSON 58: Object Detection With Speech to Text Voice Control on Raspberry Pi 5!

In this lesson, we take a massive leap forward in edge artificial intelligence and robotics by giving our Raspberry Pi 5 camera rig the gift of speech. Up until now, changing our target object of interest required editing our Python script and restarting the code. In Lesson 58, we bridge the gap between computer vision and natural human interaction by introducing local Speech-to-Text (STT) and Piper Text-to-Speech (TTS). Now, you can simply speak to your system, tell it what object you want to track, and watch as your pan-tilt camera locks on and follows your command!

Understanding the Program Architecture

The program is structured around a robust, multi-threaded architecture designed to handle audio recognition and high-speed computer vision simultaneously without lagging your video stream.

1. Voice-Driven Input via Background Threading

To ensure our camera stream maintains a smooth frame rate, we initialize a dedicated background worker thread using Python’s threading library. This thread runs independently, listening to microphone input via local STT tools.

  • When you speak your object of interest, the thread captures the audio, strips and lowercases the text, and checks for termination commands like quitting the program.

  • If a valid target is spoken, the thread pushes the string into a thread-safe communication queue and commands Piper TTS to talk back, audibly confirming your choice with a phrase like “Now Tracking”.

2. Thread-Safe Command Queuing

Because audio processing and computer vision operate on different timing loops, we use Python’s thread-safe queue mechanism. The background listening thread drops new target requests into the queue, while our main loop checks on every frame pass whether a new object of interest has arrived. This prevents race conditions and keeps our variable updates clean and reliable.

3. High-Speed OpenCV and MediaPipe Vision Loop

The main execution thread runs our primary while loop, capturing high-resolution frames from the Picamera2 camera module at 60 FPS, flipping the orientation, and feeding the frame data into MediaPipe’s efficient object detection model.

  • When detections match our active target class, the program extracts bounding box coordinates, calculates the geometric center of the object, renders visual confirmation boxes and confidence percentages on the screen, and computes pixel error offsets from the center of the display.

4. Proportional Servo Feedback Control

Once the pixel error between the target center and the screen center is calculated, our servo update function kicks in. By applying proportional gain multipliers to the horizontal and vertical error values, the script smoothly drives the SunFounder Fusion HAT servos, keeping your target locked right in the crosshairs.

Hardware Requirements

  • Raspberry Pi 5

  • Raspberry Pi Camera Module (configured with Picamera2 at 1280 by 720 resolution and 60 FPS)

  • SunFounder Fusion HAT (for driving pan and tilt servos)

  • USB Microphone (for local speech-to-text input)

  • Speaker or Headphones (for Piper text-to-speech audio feedback)

Homework Assignment for Next Week

Now that you have full voice-controlled object selection up and running, it is time for your next big challenge. For next week’s lesson, take the code provided at the bottom of this page and add mode control with three distinct operating states:

  1. find mode: In this mode, the servos remain completely switched off. The camera stays stationary, but MediaPipe actively finds, labels, and boxes your object of interest whenever it appears in the frame.

  2. track mode: In this mode, the servos are switched on, and the camera actively drives the pan-tilt rig to track and center the object of interest.

  3. scan mode: In this mode, if your target is out of view, the system actively sweeps and scans the room using the servos to find the object, and then automatically switches over to track mode once the object is located.

Grab your coffee, fire up your Raspberry Pi 5, review the complete code at the bottom of this page, and I will see you in the next lesson!

 

AI on the Edge LESSON 56: Identifying Object of Interest in Target Zone with Mediapipe

Welcome back, everyone! Paul McWhorter here from toptechboy.com, and welcome to LESSON 56 of our AI on the Edge tutorial series! Grab yourself a nice hot cup of iced coffee (no sugar needed!), sit back, relax, and let’s jump right into today’s project.

AI on the Edge LESSON 56: Identifying Object of Interest in Target Zone with MediaPipe

In our previous lessons, we learned how to use MediaPipe’s object detection task to scan our live camera feed and pick out objects using pre-trained edge AI models like efficientdet_lite0.tflite. We got our bounding boxes drawn, displayed class labels, and showed confidence scores—all running smoothly right on our Raspberry Pi!

Today, we are taking our computer vision setup to the next level by building a Target Zone Detector.

What We Are Doing Today

Instead of just blindly detecting everything in the frame, we want our edge device to actively monitor a specific region of interest—a “Target Zone”—and trigger an alert when a specific Target Class enters that boundary.

Here is what we accomplish in this lesson:

  1. Define a Fixed Target Zone: We calculate a customizable bounding box right in the center of our 1280×720 video feed (e.g., 150×250 pixels).

  2. Filter by Object Class: We set a target object (in this example, a ‘bottle’). The system ignores all other object classes detected in the scene.

  3. Calculate the Bounding Box Center: For the target object, we compute its exact center point (X_center, Y_center) on the fly and draw a tracking dot on it.

  4. Boundary & Collision Logic: We test whether the center coordinates of our target fall inside our designated target zone using the condition: zoneX1 <= X_center <= zoneX2 and zoneY1 <= Y_center <= zoneY2.

  5. Dynamic Visual Feedback:

    • Idle State: When no target is inside the zone, the box remains Green.

    • Active State: The moment the target’s center point steps into the target zone, the boundary box turns Red, and an alert banner—”Target Has Entered the Zone”—is dynamically rendered on the camera feed.

  6. Real-Time Performance: We track frame rate (FPS) using an exponential moving average calculation to keep an eye on edge processing speed.

Homework / Challenge Assignment

Now that you have seen how to set up logic for a single target object in a single zone, here is your homework assignment to solidify what you’ve learned!

  1. We want to now actively track the selected Object of Interest. Turn your servos on on the pan tilt camera, and keep the Object of Interest in the center of the camera view.

Give it a try on your own before watching the solution in the next video. Remember: You don’t learn by watching; you learn by doing!

Discussion & Community

Did your target zone detection work smoothly on your Raspberry Pi setup? How many FPS are you getting with your Pi Camera? Leave a comment below on YouTube to share your results, questions, or modifications!

If you enjoyed this tutorial, hit that Like button, Subscribe to the channel, and share it with anyone who wants to learn real-world AI on the Edge.

As always, I’ll see you guys in the next lesson!

General Knowledge / Context

  • MediaPipe Tasks API: MediaPipe Python Tasks API provides high-level Python interfaces for object detection, landmasking, and classification tasks utilizing lightweight edge models such as TensorFlow Lite (.tflite).

  • PiCamera2 & OpenCV: In modern Raspberry Pi OS (Bullseye/Bookworm), Picamera2 handles native camera hardware capture in raw formats like RGB888, which can then be converted to MediaPipe Image objects (mp.Image) and rendered on OpenCV (cv2) GUI windows seamlessly.

 

AI on the Edge LESSON 49: Live Animated Avatars on the SSD1306 OLED With OpenCV

Hey everybody! Welcome back to the AI on the Edge series!

In Lesson 49, we’re doing something that’s just plain cool — we’re turning that tiny little SSD1306 OLED into a live animated avatar that follows your face and hands in real time!

That’s right — no more static images, and no more being tethered to your desktop and monitor. This time your OLED is alive! Move your head, smile, raise your eyebrows, wave your hands… and the little display instantly mirrors your movements like a miniature digital twin. It’s like having a tiny version of yourself living on that 128×64 screen!

What You’ll Learn in This Lesson:

  • How to use MediaPipe Face Mesh to track 468 facial landmarks
  • How to combine it with Hand Tracking at the same time
  • The trick to projecting those landmarks onto a tiny OLED in real time
  • How to draw both points and connecting lines to create smooth, animated avatars
  • How to keep everything running fast on the Raspberry Pi

This is one of those projects that makes people stop and say “Wait… how is that even possible on that little screen?!”

By the end of this lesson, you’ll have a foundation for building all kinds of fun real-time avatar projects — everything from simple face mirrors to gesture-controlled characters and beyond.

Why This Is Awesome:

We’re not just displaying video on the big screen anymore — we’re compressing all that AI power down onto a cheap little OLED. This is real AI on the Edge stuff, and it looks incredibly impressive for how simple the hardware is.

Whether you want to build interactive displays, robot faces, wearable tech, or just blow your friends’ minds, this lesson gives you the core technique to make it happen.


Ready to make your OLED come alive?

Fire up the code, run it, and watch your tiny digital self start moving with you. Then start experimenting! Try drawing just the eyes and mouth, or just the hands, or maybe even turn it into a little stick figure avatar.

As always, I want to see what you create with this! Drop your versions in the comments or tag me — I love seeing the creative stuff you guys come up with. Don’t just copy and paste my code. Show you really understand the video by making the code yout own.

Enough talk, lets get this party started!

 

AI on the Edge LESSON 48: Hand Detection in OpenCV and MediaPipe on the Raspberry Pi

In Lesson 48 we’re taking a huge step forward — we’re bringing real-time hand detection to the Raspberry Pi using the incredible power of MediaPipe combined with OpenCV and the Pi Camera. This is one of those projects that feels like pure magic when you see it working: the camera picks up your hands instantly, tracks all 21 landmarks on each hand, draws beautiful connections in real time, and does it all right on the edge with no cloud required.
You’ll learn how to set up MediaPipe’s Hands solution for reliable multi-hand tracking, how to efficiently process frames from the Picamera2 library, flip and convert images for proper display, and draw professional-looking landmarks and connections with custom styling. We also keep a smooth FPS counter running so you can see exactly how well your Pi is performing.
This lesson is exciting because hand tracking is the foundation for so many advanced gesture-control projects — think touchless interfaces, sign language recognition, robotic control, virtual instruments, and interactive art installations. Once you have solid hand detection running smoothly on your Raspberry Pi, the creative possibilities are almost endless.
So grab your Raspberry Pi, fire up the camera, and let’s get those hands dancing on the screen! As always, the full code is waiting for you below. Watch the video, follow along, and then start experimenting — I can’t wait to see what you build with this!
Let’s make something awesome!

 

AI on the Edge LESSON 46: Ultimate Dazzleing Running Rainbow On a NeoPixel Ring

In this lesson we create one of the most beautiful and satisfying NeoPixel effects — a smooth, continuous, running rainbow on a 12-LED ring. We affectionately have named this pattern the “Runbow”. This is the “ultimate” version because it is both visually stunning and highly educational, as we explore two fundamentally different programming approaches the Runbow.
The first method is the most intuitive: we place a fixed rainbow across the ring (each LED gets a different hue equally spaced around the color wheel) and then simply rotate the entire pattern one position at a time. This approach is easy to understand because it mimics physically moving a colorful wheel. We save the color of the first pixel, shift all the other pixels one place, and then place the saved color at the end. It feels like a conveyor belt of color circling around the ring.
The second method is more elegant and mathematically pure. Instead of storing and shifting colors, we recalculate the color of every LED on every frame using a moving offset. For each LED we compute its hue as (i / LED_COUNT + offset) % 1.0, where i is the LED’s position and offset is a value that slowly increases over time. This creates a perfectly smooth rainbow that flows around the ring without ever shifting raw RGB values. Because we regenerate the pattern fresh each cycle, there is no risk of color corruption or accumulated errors.
Both techniques produce a gorgeous running rainbow, but they teach very different programming mindsets. The first method helps you deeply understand array manipulation and data movement. The second method introduces the powerful concept of using mathematics and offsets to create motion — a technique used frequently in advanced LED animations, games, and visual effects. In the end, the offset method tends to look smoother and is easier to extend with additional effects (such as brightness pulsing), but both approaches are valuable skills for any embedded AI or IoT developer working with addressable LEDs.You can adjust the speed of the rainbow by changing how much you increment the offset each loop or by modifying the delay. Once you master these two methods, you will be able to create almost any animated pattern you can imagine on your NeoPixel ring. Lets get this party started!
Here is the code we developed in this video:

This is the schematic we are using to connect the NeoPixel ring:

NeoPixel
NeoPixel Schematic