Tag Archives: openCV

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 44: Displaying Live MediaPipe FaceMesh Avatar on SSD1306 OLED in OpenCV

This is Getting Crazy Cool!

Hey guys, welcome back to the Fusion AI Lab series, AI on the Edge! In Lesson 44 we just took things up another notch. We’re now running MediaPipe Face Mesh on our Raspberry Pi 5, pulling all 468 facial landmarks in real time, and then drawing a clean, smooth face contour directly onto a tiny 128×64 OLED display!

That’s right — your face is now living on that little monochrome OLED in real time! Every eyebrow raise, smile, head tilt, and eye movement gets faithfully reproduced on the display. We’re pulling the camera feed with picamera2, processing it with MediaPipe’s powerful face mesh, scaling the landmarks down to OLED resolution, and then drawing the official FACEMESH_CONTOURS connections using PIL. The result is surprisingly smooth and fun to watch! This project really shows the power of combining modern AI vision tools with simple embedded hardware. Watching your own face rendered in real time on a tiny OLED is just plain awesome — it feels like sci-fi stuff from just a few years ago, and now we’re doing it on the Edge with our Fusion AI Lab kit!

If you’ve been following along, you’re starting to see how powerful these tools are becoming. We’re no longer just detecting faces — we’re understanding the structure of the face and visualizing it however we want. And we’re just getting started!

So fire up your Raspberry Pi, grab that OLED, and let’s keep pushing the limits. You’re doing some seriously cool stuff!

In this lesson we are still using our standard Fusion AI Lab Kit cirtuit from the earlier lessons:

Fusion Hat Circuit Diagram
This is the circuit we will use moving forward in the class

We have also connected the OLED to the above circuit according to this diagram:

OLED
SSD1306 OLED Connected to the Fusion AI Hat
AI On the Edge, Raspberry Pi

AI on the Edge LESSON 42: Create Composite Images Using Masks in OpenCV and MediaPipe

In this exciting project, we combine a Raspberry Pi 5, the Fusion AI Lab Kit, a Pi Camera, and a remote IP camera to generate a stunning real-time composite video. Watch as a glowing, translucent MediaPipe face mesh of my face hovers magically over live video of the Mighty River Nice scenery captured by an IP camera. The effect looks futuristic and professional — perfect for creative video effects, interactive installations, or just blowing your mind with computer vision! Using Picamera2 for high-frame-rate local capture and OpenCV with an RTSP stream from the river camera, we process everything in real time. MediaPipe’s Face Mesh detects and tracks facial landmarks, which we draw as beautiful cyan/teal contours with glowing irises. Then we create a clean mask, separate the mesh foreground from the river background, and blend them seamlessly into one composite frame. You’ll see every debugging layer live on screen too — meshLayer, mask, inverted mask, riverBG, and meshFG — so you can understand exactly how the magic happens.This tutorial is beginner-to-intermediate friendly and packed with practical OpenCV + MediaPipe techniques you can adapt for your own augmented reality projects. Whether you’re a longtime follower of the Paul McWhorter channel or new to the Fusion AI Kit, you’ll walk away inspired and ready to build your own hovering effects, overlays, or interactive displays.Grab the full code from the video description, fire up your Pi 5, and start creating jaw-dropping computer vision projects today. Drop a comment and let me know what you’d like to overlay next — another face mesh, hand tracking, or something completely different? Let’s keep pushing the limits of what we can do with affordable AI hardware!

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 41: Creating FaceMesh Using MediaPipe in OpenCV

In this project, I demonstrate how to create a smooth, real-time face mesh overlay using the Raspberry Pi 5, the official Pi Camera, MediaPipe, and OpenCV. The program captures live video from the camera and draws a detailed, colorful mesh that follows every movement of the face with high accuracy. The result is a visually appealing augmented reality-style effect that runs efficiently even on a single-board computer.

The goal of this project is to build a responsive face tracking system that detects and draws 468 facial landmarks in real time. This creates a striking mesh that highlights the contours of the face, eyes, lips, and jawline, making it an excellent foundation for more advanced computer vision projects like virtual filters, AR effects, or interactive installations.

The program follows a straightforward but efficient real-time vision pipeline. First, it initializes the Raspberry Pi Camera using the modern picamera2 library, configured for 1280×720 resolution at 60 frames per second. It then sets up MediaPipe’s Face Mesh solution with landmark refinement enabled for better eye tracking.

In the main loop, the program continuously grabs a frame from the camera, corrects its orientation, and converts it from BGR to RGB format since MediaPipe expects RGB input. The frame is then passed to the Face Mesh model for processing. When a face is detected, the program draws multiple layers of graphics on top of the image: a fine tesselation mesh across the entire face, thick and vibrant contours around the major facial features, and special highlighting on the irises. Finally, the processed frame is displayed in an OpenCV window, creating a smooth and engaging real-time visualization.

This approach works particularly well on the Raspberry Pi 5 because it balances visual quality with performance. By limiting detection to a single face and using efficient drawing methods, the application maintains high frame rates while producing a professional-looking result. The multi-layer drawing technique (tesselation + contours + irises) gives the mesh depth and visual appeal that single-pass drawings often lack.

The project makes use of several powerful technologies: picamera2 for fast camera access, Google’s MediaPipe for high-speed machine learning-based landmark detection, OpenCV for image handling and display, and NumPy for efficient array operations.

This face mesh project serves as an excellent stepping stone into real-time AI and computer vision on embedded hardware. Once you have the basic mesh working, it becomes much easier to expand into creative applications such as face filters, gesture recognition, or overlaying the mesh onto other video sources.

The code developed in the video lesson is presented below:

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 39: Understanding MediaPipe Data Structures

In this video lesson I show you how to understand the data structures returned by MediaPipe. I show you how to peel the data structure back, to get at the useful information.

When you run face detection with MediaPipe, the results object it returns is not a normal dictionary or list. It is a special custom object called SolutionOutputs. The easiest way to explore it is to start by checking the main attribute: results.detections. This is a Python list that contains one entry for every face detected in the current frame. If no faces are found, results.detections will be None or an empty list.

To extract useful information, you loop through results.detections. Each item in that list is a Detection object. From this object, you can access two main things: the confidence score using detection.score[0], and the location data using detection.location_data. Inside location_data, you will find relative_bounding_box (which gives you xmin, ymin, width, and height as values between 0 and 1) and relative_keypoints (a list of 6 facial points such as eyes, nose, and mouth).

The standard method is to first get the frame’s height and width, then multiply the normalized values (like xmin and width) by the actual pixel dimensions of the image to convert them into usable pixel coordinates. You can then use these coordinates with OpenCV functions such as cv2.rectangle() for the box or cv2.circle() for the keypoints.

By using simple print(type()), print(dir()), and print() statements on results, results.detections, and individual detection objects, you can quickly discover the full structure. This step-by-step approach — starting from results → detections → individual detection → location_data — lets you reliably reach all the useful information MediaPipe provides.

Below is the code we developed in the video.

 

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 37: Using RTSP and IP Cameras in OpenCV on Raspberry Pi 5

The code below shows the work we did in this lesson.

AI on the Edge Lesson 37: Using RTSP and IP Cameras in OpenCV on Raspberry Pi 5

Hey guys! Welcome back to our AI on the Edge series. In our previous lessons, we’ve had a blast working with standard USB webcams, but if you are building a real-world computer vision application, an automation rig, or a security monitoring setup around your home or farm, USB cables just aren’t going to cut it. You need to pull video feeds from remote IP cameras using the Real-Time Streaming Protocol (RTSP).

Today, we are taking that exact step on the Raspberry Pi 5, connecting to an IP camera, streaming the feed smoothly into OpenCV, and—most importantly—solving the dreaded latency problem that plagues RTSP feeds.

The Big Challenge: Conquering RTSP Latency

If you’ve ever tried pulling an RTSP stream into OpenCV straight out of the box, you’ve probably noticed something frustrating: the video lags behind real-time, sometimes by several seconds or even tens of seconds.

Why does that happen? Because by default, FFmpeg and OpenCV buffer incoming frames to ensure smooth playback. But when you are doing computer vision, AI inferencing, or real-time tracking on the edge, you don’t want old history—you want right now.

To fix that, we pass the cv2.CAP_FFMPEG backend flag and immediately flush the buffer by setting the property to 0. This forces OpenCV to drop the backlog and grab the absolute newest frame available from the camera stream, keeping your Pi 5 processing live data in real-time.

Understanding the Script Structure

Let’s break down the key parts of today’s implementation:

  • Credentials & Resolution: We import a separate secret file to keep our camera IP addresses, usernames, and passwords safe and out of public repositories. We lock our resolution at 1280×720 to balance crisp detail with the Pi 5’s processing overhead.
  • Smooth FPS Calculation: Instead of a jittery raw frame-rate readout, we use an exponential moving average to give us a stable, readable performance metric on screen.
  • The Display Window: We configure a GUI window using OpenCV’s window flags so we can easily position and resize our output feed on the desktop.

Drop Your Questions Below

Working with network streams can sometimes be tricky depending on your specific camera’s firmware, codec settings, and network stability. If you run into any connection drops or lag spikes on your Raspberry Pi 5, drop a comment on the video!

Keep building, stay creative, and I will see you guys in Lesson 38!

Here is the code developed in the video