Tag Archives: AI

AI on the Edge LESSON 57: Detecting and Tracking Objects with MediaPipe

Welcome back, my friends! This is Paul McWhorter from toptechboy.com, and welcome to LESSON 57 of our AI on the Edge tutorial series.

In this lesson, we take what we’ve learned about local object detection using MediaPipe and put it to work in the physical world! We are building an active target tracking system using a 2-DOF pan-tilt servo rig driven by the Raspberry Pi and the Fusion HAT.

By calculating the positional error between the center of our targeted object (in this case, a bottle) and the center of our frame, we compute feedback adjustments to continuously drive our pan and tilt servos—keeping the target locked right in the center of the camera view.

What You Will Learn in Lesson 57

  1. Setting Up MediaPipe Object Detection: Configuring the efficientdet_lite0.tflite model with threshold options and maximum result limits.

  2. Integrating Picamera2 with OpenCV: Capturing high-resolution camera feeds (1280 by 720 at 60 FPS) and converting raw frames to MediaPipe-compatible image formats.

  3. Bounding Box Analysis & Centroid Calculation: Extracting bounding box dimensions and computing the precise pixel center of detected targets.

  4. Proportional Error Servo Control: Using gain coefficients to convert pixel displacements into physical angle corrections for pan and tilt servos.

  5. Deadband Implementation: Preventing servo chatter and hunting by defining a deadband region around the center of the screen.

Hardware Requirements

  • Raspberry Pi 5 (or Raspberry Pi 4)

  • Raspberry Pi Camera Module (configured with Picamera2)

  • SunFounder Fusion HAT (or compatible servo controller HAT)

  • 2-DOF Pan-Tilt Servo Kit (Pan attached to Pin 2, Tilt attached to Pin 3)

  • Target Object (e.g., a water bottle or any object recognized by COCO dataset categories)

How the Proportional Tracking Control Works

To keep the target centered in the video frame, we continuously measure the distance in pixels between the frame’s center and the center of the target’s bounding box:

errorX = boxCenterX – screenCenterX

errorY = boxCenterY – screenCenterY

If the magnitude of the error exceeds our deadBand limit (15 pixels), we adjust the pan and tilt angles proportionally:

panAngle = panAngle – (errorX * panGain)

tiltAngle = tiltAngle + (errorY * tiltGain)

  • Pan Limits: Clamped between -90 and +90 degrees.

  • Tilt Limits: Clamped between -45 and +45 degrees.

Homework Assignment

Now that you have basic single-target tracking running smoothly:

  1. Multi-Target Priority: Modify the script so that if multiple target objects appear on screen, the camera locks onto the object closest to the center of the frame rather than defaulting to the first detected bounding box.

  2. Dynamic Smoothness Filter: Add an exponential moving average or low-pass filter to the bounding box center coordinates to smooth out tracking jitter when the object moves rapidly across the field of view.

  3. Audibly speak the Object of Interst to track. This will require you to incorporate Speech to Text voice recognition.

Get your coffee, fire up your Raspberry Pi, copy the code above into your IDE, and let’s get tracking!

General Artificial Intelligence & Edge Computing Concepts

What is Edge AI?

Edge AI refers to the deployment of artificial intelligence algorithms directly on hardware devices, such as microcontrollers, single-board computers, or edge accelerators, located close to the source of data generation rather than relying on cloud-based processing servers.

Key Advantages of Local Inference:

  • Low Latency: Processing video feeds directly on an edge device eliminates network transmission delay, enabling real-time feedback loops critical for robotics and physical pan-tilt tracking.

  • Privacy & Security: Sensitive visual or audio data remains on the local hardware without requiring continuous streaming to third-party cloud infrastructure.

  • Network Independence: Edge deployments operate reliably in remote or bandwidth-constrained environments without requiring active internet connectivity.

  • Cost Efficiency: Running localized models eliminates recurring per-API-call cloud infrastructure and compute costs.

Proportional-Integral-Derivative Control Concepts:

  • Proportional Control: Calculates an output value proportional to the current error magnitude. High proportional gain results in aggressive responses, while low gain yields sluggish tracking.

  • Deadband: A non-responsive range around the zero-error position where control adjustments are intentionally suspended to prevent motor oscillation, noise amplification, and mechanical wear.

 

AI on the Edge LESSON 55: Object Detection on the Raspberry Pi 5 With MediaPipe

Welcome back, everyone! This is Paul McWhorter from TopTechBoy.com, and welcome to Lesson 55 in our AI on the Edge tutorial series!

If you’ve been following along, grab yourself a nice, hot cup of black coffee, fire up your Raspberry Pi 5, and let’s get right into it!

What We Are Doing in Lesson 55

Up to this point in the series, we have spent a lot of time working with land-marking models—tracking human hands, pose points, and facial features. But today, we step into full-blown computer vision object detection!

In this lesson, we learn how to implement MediaPipe’s lightweight, standalone object detection task API (mediapipe.tasks.python.vision.ObjectDetector) running on the Raspberry Pi 5. Using the pre-trained COCO dataset model (efficientdet_lite0.tflite), our Pi 5 can instantly identify up to 80 different real-world objects—from people, cups, and bottles to chairs, laptops, and cell phones—and draw real-time bounding boxes around them on our video feed.

Key Takeaways & Concepts Covered

  1. Setting Up the MediaPipe Object Detector Task API:

    Unlike older mp.solutions wrappers, the modern MediaPipe Tasks API requires configuring BaseOptions with the explicit local path to your .tflite model asset (modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite").

  2. Configuring Detector Parameters:

    • score_threshold (0.35): Sets the minimum confidence score required to accept a detected object. Lowering this value captures weaker detections but risks false positives; raising it ensures high-confidence target lock.

    • max_results (3): Limits the maximum number of bounding boxes predicted per frame to keep our processing loop lean and fast.

  3. Decoding Detection Bounding Boxes:

    MediaPipe returns detected objects with bounding box geometry relative to the frame origin (origin_x, origin_y, width, height). We parse these dimensions directly into OpenCV integers (x, y, w, h) to draw bounding boxes with cv2.rectangle().

  4. Category Labeling & Confidence Rendering:

    We pull category names (category_name) and classification scores (category.score) from detection.categories[0], format the confidence as a percentage, and render clean text labels anchored directly above each target box.

  5. Optimal Frame Alignment & FPS Monitoring:

    We capture 60 FPS video directly from picamera2 in RGB888 format, flip it properly to match our physical camera mounting, and track real-time frames-per-second using our low-pass smoothing formula.

What You Will Need

To follow along with this lesson, ensure you have your standard gear ready from the kit:

  • Raspberry Pi 5 (with Active Cooler/Heat Sink)

  • Raspberry Pi Camera Module 3 (or compatible Camera Module driven via picamera2)

  • SunFounder Fusion AI Lab Kit (with pre-downloaded MediaPipe .tflite model weights)

Homework Assignment

Alright, class, you know I never let you off the hook without some homework!

Your assignment for this lesson is to take this foundational object detection script and upgrade its intelligence:

  1. Target Filter: Modify the loop so that it only draws bounding boxes around a specific target object of your choice (for instance, only detect "person" or only detect "bottle"), ignoring all other detected COCO classes.

  2. Target Tracking Dot: Calculate the exact center pixel coordinate $(x + \frac{w}{2}, y + \frac{h}{2})$ of your targeted object, and draw a solid red target circle (cv2.circle) right on the object’s center point.

  3. Dynamic On-Screen Target Status: If your target object is visible in the frame, display a green text banner in the upper corner reading "TARGET ACQUIRED". If it leaves the frame, change the status banner to a red "SEARCHING...".

Give it a shot, take your time, and write your solution from scratch without copying and pasting! Drop a comment on the YouTube video to let me know if you got it working or if you ran into any snags.

Until next time, get your coffee, keep coding, and I’ll see you guys in Lesson 56!

 

AI on the Edge LESSON 54: Custom Hand Gesture Recognition & Dynamic NeoPixel Control

Welcome back, everyone! In this lesson, we take edge AI interactivity to the next level. Up until now, we have learned how to track hand landmarks, extract pixel coordinates, and calculate distances using MediaPipe. Today, we put all those building blocks together into a self-contained, real-time hand gesture recognition system running on the Raspberry Pi 5.

We are not relying on a black-box pre-trained gesture model. Instead, we write our own mathematical feature extractor, train our custom hand poses on the fly, save those gesture profiles directly to disk, and map each gesture to a non-blocking lighting animation on a NeoPixel LED ring driven over SPI.

System Architecture Overview

The system operates in three distinct phases running simultaneously inside our main loop:

  1. Normalized Feature Extraction: MediaPipe extracts 3D hand landmark coordinates. To make gesture detection scale-invariant and distance-invariant, we normalize the distances between the fingertips and the wrist against a reference palm dimension.

  2. Nearest-Neighbor Gesture Matching: Live distance vectors are compared against stored gesture profiles using a mean absolute error algorithm. If the error falls below our maximum threshold, the gesture is identified; otherwise, it defaults to “Unknown.”

  3. Non-Blocking Hardware Animation Dispatch: The recognized gesture string triggers corresponding pattern functions on the 12-LED NeoPixel ring via neopixel_spi. Animations use non-blocking time delta checks to preserve high camera preview frame rates.

Gesture-to-Hardware Animation Mapping

Our script contains predefined animation routines mapped to recognized gesture strings. Here is how each pose controls the LED ring:

Recognized Gesture LED Animation Pattern Visual Effect
fist Red Pulse Sinusoidal breathing effect fading in and out
palm Solid Beacon Full brightness white ring
peace Dual Spinner Dual green and blue dots rotating over a dim red background
thumbsup Solid Success Constant green feedback
point Radar Sweep A green trailing dot chasing around a red ring
rock Cyan Flame Sinusoidal green-cyan intensity modulation
ok Rainbow Wheel Rotating HSV spectrum cycle
Unknown / Unmatched Clear LEDs turn completely off

Interactive Keyboard Controls

While the camera loop is active, you can interact with the system using your keyboard in the preview window:

  • Spacebar: Captures the current hand pose’s normalized distance vector, assigns it to the active target gesture name, and saves the updated dataset to gestureNEOPIXEL.pkl.

  • t: Prompts the terminal to add a new custom gesture name to your training queue dynamically.

  • d: Prompts the terminal to delete a specific gesture profile from the saved dataset.

  • q: Clears the NeoPixel ring, closes the camera pipeline, saves training data, and terminates the program cleanly.

Hardware Wiring Reference

Ensure your Raspberry Pi 5 SPI interface is enabled before running the code (sudo raspi-config -> Interface Options -> SPI).

  • NeoPixel Data Input (DIN): Connect to GPIO 10 (SPI0 MOSI / Pin 19).

  • NeoPixel Ground: Connect to any GND pin on the Pi expansion header.

  • NeoPixel Power: Connect to an external 5V power supply (ensure a shared ground with the Raspberry Pi).

Homework Challenge

Now that you have a working gesture recognition system linked to hardware animations, try adding your own custom gestures! Train a brand-new gesture pose (like a “three-finger count”), write a corresponding non-blocking animation function in updateNeoPixels(), and share your video results.

Here is the code we developed in the video lesson:

Homework & Next Steps

Try setting up the circuit and running the code on your Raspberry Pi 5. Observe how the frame rate behaves when driving both the camera preview, the hand-tracking model, and the I2C OLED display simultaneously. For your homework, extend this script to include a dynamic color slider or gesture-based brightness control!

AI on the Edge LESSON 51: Gesture Control of NeoPixel Ring

 

Welcome back, everyone! In today’s class, AI on the Edge LESSON 51: Gesture Control of NeoPixel Ring, we are bridging the gap between computer vision and physical hardware in a big way. Up until now, we have spent a lot of time tracking hands, plotting 3D landmarks, and rendering graphical user interfaces directly onto our camera feeds. But today, we take those virtual interactions and project them out into the real physical world using a 12-LED NeoPixel ring driven over SPI by our Raspberry Pi 5.

The core concept of this lesson is creating an intuitive, spatial gesture interface. Imagine looking at your monitor display where a graphical representation of a circular LED ring is overlaid onto your live camera feed. By extending your hand and placing your index finger tip over any of the twelve virtual LED nodes on screen, the system detects your hover position. When you pinch your thumb and index finger together, the program registers a toggle event, turning the corresponding physical LED on or off on the actual hardware ring sitting on your workbench.

To make this work reliably without wild bouncing or rapid flickering, we dive deep into the mechanics of gesture latching and debouncing. When a student first attempts spatial tracking, holding a pinch gesture over a target often causes the state to flip back and forth dozens of times per second. In this lesson, we implement a stateful boolean latch flag that locks down the trigger state the moment a pinch is registered. The algorithm strictly requires you to release the pinch—breaking contact between your thumb and index finger—before it will accept another command. This simple state machine logic is essential for turning noisy computer vision inputs into rock-solid physical control interfaces.

Beyond individual pixel control, we also construct a dynamic UI Mode Selection Panel on the right side of the screen. By hovering over the panel buttons and executing the same pinch gesture, you can cycle through five distinct lighting animation profiles in real time:

  • Static Mode: Displays fixed, custom-assigned palette colors for each active LED node.
  • Rainbow Mode: Distributes a static 360-degree color wheel spectrum across all twelve positions using HSV-to-RGB conversion algorithms.
  • Pulsate Mode: Modulates the physical brightness using a squared sine wave math function to guarantee a true black-floor fade on turned-on pixels.
  • Blink Mode: Synchronizes an on/off strobe animation derived from live system timestamps.
  • Running Rainbow Mode: Dynamically rotates the active hue spectrum around the ring by stepping the color offset frame-by-frame.

We handle all of this frame rate processing seamlessly using high-speed SPI bus communications through neopixel_spi and direct frame acquisition via Picamera2. By keeping auto-write disabled until all mathematical animation steps and hand-tracking calculations are completed for a given frame, we push smooth, flicker-free updates out to both the physical hardware and the OpenCV display overlay simultaneously.

Work through the complete Python script provided on this page, load it onto your Pi, and assemble your hardware circuit. Pay close attention to how the spatial math maps coordinates from normalized MediaPipe landmark spaces directly into circular screen coordinates using polar angle formulas. Grab your components, fire up your IDE, and let’s get building!

 

 

AI on the Edge LESSON 50: Control NeoPixel Ring With Hand Gestures and MediaPipe

Hey everybody, welcome back to AI on the Edge! In Lesson 50 we’re taking things to the next level — we’re combining real-time hand tracking with a physical NeoPixel ring so you can change colors just by pointing at the screen! Using MediaPipe on the Raspberry Pi, we track your index finger tip and turn the big camera view into an interactive color picker. Hover over any of the seven big colored circles (Red, Green, Blue, Cyan, Magenta, Yellow, or Black) and — boom — the NeoPixel ring instantly lights up with that color!
This lesson is super fun because it finally brings together everything we’ve been building: smooth hand detection, coordinate mapping between camera and OLED, visual feedback, and real-world hardware control. You’ll see your finger tracked with a big blue dot, watch the selected color update live on both the screen and the OLED, and control actual RGB lights with nothing but a gesture.
By the end of this lesson you’ll have a working gesture-controlled color mixer that feels like real interactive magic. This is the kind of project that makes people say “Wow, you built that on a Raspberry Pi?!”
This is also a perfect foundation for even cooler future projects — gesture-controlled lighting, touchless interfaces, interactive games, or even a full gesture-controlled robot or light show.
So grab your Raspberry Pi, hook up that NeoPixel ring, fire up the camera, and let’s start waving our hands around like we’re casting spells! As always, the complete code is right below the video. Watch along, run the code, then start tweaking it — maybe add more colors, make the ring react differently to gestures, or combine it with the hand connection drawing from earlier lessons.I’m really excited to see what you create with this one!Now go make something awesome! 

This is the core circuit we use for the class:

Fusion Hat Circuit Diagram
This is the circuit we will use moving forward in the class

This is the schematic to add the OLED:

OLED
SSD1306 OLED Connected to the Fusion AI Hat

Then this is the schematic to add the NeoPixel

NeoPixel
NeoPixel Schematic