AI On the Edge, Raspberry Pi

AI on the Edge LESSON 31: Facial Recognition in OpenCV Using Haarcascades

Hey everyone, this is Paul McWhorter from toptechboy.com. Welcome back to our AI on the Edge tutorial series. If you’ve made it this short distance through the course, you are doing fantastic.

Today, we are stepping away from simply pulling a clean, high-frame-rate video stream off our hardware, and we are finally going to start doing some real Computer Vision. We are going to teach our machine how to look at an image, process it, and actually locate human faces in real time.

Go ahead and grab yourself a hot cup of coffee. Turn off your cell phone, close your other browser tabs, and let’s get ready to write some serious engineering code.

The Core Concept: What is a Haar Cascade?

Up until now, we’ve been focused on the plumbing—getting the camera configured, setting up the dimensions, and flipping the frames so they look right on our screens. Today, we introduce the Haar Cascade Classifier.

Think of a Haar Cascade as an incredibly smart, pre-trained statistical filter. Decades ago, researchers realized that human faces share universal geometric patterns of light and dark. For example:

  • The bridge of your nose is almost always brighter than the shadows on either side of it.

  • Your eye sockets are consistently darker than your forehead or your cheekbones.

OpenCV comes pre-packaged with these mathematical descriptions of a face. The algorithm works by taking a tiny “sliding window” and dragging it across your video frame pixel by pixel, looking for those specific arrangements of light and dark gradients. When it finds a cluster that matches the math, it flags it as a face.

Why Grayscale Matters in Machine Learning

If you look closely at our pipeline logic for today, the absolute first thing we do after capturing a raw frame from our camera is strip away all the color and convert the image to grayscale.

Why do we do this? Color is a computational luxury we cannot afford when doing real-time edge processing. To a computer, a color image consists of three separate channels: Red, Green, and Blue. That means for a standard resolution, the processor has to crunch three times the data.

Because Haar Cascades only care about the contrast—the relationship between light and dark areas—the actual color of your skin, your hair, or your shirt is completely irrelevant. By dropping the frame down to a single grayscale intensity channel, we cut our processor’s workload by 66% instantly. This is how we keep our edge hardware running lightning-fast without lagging the video feed.

Tuning the Detection Engine

When we tell OpenCV to look for faces using the detectMultiScale function, we pass three critical parameters that you need to master. If you don’t tune these right, your program will either miss faces entirely or start thinking your coffee mug or a pattern on the wall is a human being.

  • scaleFactor=1.1: A single Haar Cascade model is trained at a fixed size. But a face could be right next to the lens (huge) or all the way across the room (tiny). The scaleFactor tells OpenCV to shrink the image by 10% on each pass, creating a “layer cake” of images at different scales so the sliding window can catch faces of any size.

  • minNeighbors=5: As the sliding window moves, it might find dozens of potential matches around your eyes and nose. This parameter states that a face must be detected in at least 5 overlapping bounding boxes before the system officially declares, “Yes, that is a human face.” Raising this number reduces false positives but makes the system less sensitive.

  • minSize=(60,60): This tells the algorithm to completely ignore any detected objects smaller than a 60×60 pixel block. This prevents the system from wasting CPU cycles trying to analyze tiny bits of background noise in the distance.

The Secret to a Smooth FPS Counter

You will notice a very specific mathematical formula used to calculate our Frames Per Second (FPS) in this lesson. If you simply calculate $1 / \Delta T$ every single frame, your FPS display will jitter wildly on screen, flashing unreadable numbers back and forth because individual frames might take a millisecond more or less to process.

To fix this, we implement a digital Low-Pass Filter.

Every loop, we take 95% of our previous FPS value and add only 5% of our current instantaneous speed measurement. This creates a beautiful, smooth running average that responds immediately to system slowdowns but remains steady and completely legible on the screen. It is an elegant engineering solution to a common UI headache.

Homework Assignment!

You didn’t think you were going to get out of here without homework, did you? No shortcuts in this classroom!

Now that your program can successfully locate faces and draw a bounding box around them, it’s time to take it to the next level. Your assignment is to modify this program to isolate and track only the single largest face in the frame.

If multiple people walk into the camera’s view, your script must evaluate the dimensions of the returned bounding boxes, determine which person is closest to the lens (the largest box), and draw a bright green rectangle around only that leader face, while ignoring everyone else.

Here is the code developed in today’s lesson.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

Welcome, Makers!

Well, hello there! It is absolutely fantastic to have you back. I’m Paul McWhorter, and today, we are taking a massive step forward in our AI on the Edge journey.

Up until now, we’ve been hard-coding our color thresholds (those pesky Lower Color and Upper Color values) to tell our camera what to look for. That’s fine for a science experiment, but it’s not exactly “smart,” is it? If the lighting changes, or if we want to track a different colored object, we have to go back into the code and manually edit those numbers.

Not anymore!

In today’s lesson, we are building a tool that lets us teach the AI. We’re going to use the mouse to draw a Region of Interest (ROI) right on our camera feed. The system will look at the pixels inside that box, calculate the average Hue, Saturation, and Value, and automatically set our tracking range for us.

This is the kind of professional-level functionality that turns a hobby project into a true, intelligent machine.

The Concept: From Hard-Coding to Dynamic Learning

The magic happens in our mouseAction function. Instead of just reading pixel values, we are now implementing a “click-and-drag” system:

  1. Click and Hold: We capture the startX and startY coordinates.

  2. Drag: We draw a rectangle in real-time so we can see exactly what area we are selecting.

  3. Release: We take that specific slice of the image, convert it to the HSV color space, and use the cv2.mean() function to find the average color properties.

  4. Auto-Tune: We set our LC (Lower Color) and UC (Upper Color) based on that average.

By doing this, the system learns what “object” we want to track on the fly. It’s elegant, it’s powerful, and it feels like real magic when you see those servos snap onto your target after a quick mouse drag.

What We’ve Accomplished

By the end of this lesson, you will have a system that:

  • Visually selects an object using the mouse.

  • Automatically calculates the optimal HSV thresholds for that specific object.

  • Updates the tracking behavior immediately without needing to stop or re-run the code.

  • Maintains that professional “Edge” feel, giving you real-time feedback on your FPS and mouse position data.

A Note on the “Edge”

Remember, we aren’t just running code; we are running on hardware. When we calculate the mean of the ROI, we are doing real image processing on the fly. You’ll notice the Composite and Mask windows updated immediately, giving you a visual confirmation that your “teacher” (you!) has successfully guided the “student” (the AI).

This is the power of working with OpenCV and the Raspberry Pi. You are building a system that observes, thinks, and reacts—all in real-time.

Get Ready to Build

Grab your Pi, make sure your servos are ready to go, and let’s get that camera calibrated. You’ve put in the work to get this far, and today is where all that effort starts to feel really rewarding.

I’m incredibly proud of how far you’ve come. Let’s dive in and start building!

Are you ready to see how accurately your Pi can “see” once you’ve given it the ability to learn from your selections?

In the video lesson we developed the following code.

 

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 29: Improved Proportional Object Tracking with Pan Tilt Camera

AI on the Edge LESSON 29: Improved Proportional Object Tracking with Pan Tilt Camera

Hey everyone, Paul McWhorter here from TopTechBoy.com. Welcome back to our channel, where we learn to build real, intelligent systems on edge hardware. Go ahead and grab yourself a nice hot cup of coffee or a big glass of iced tea, because today we are going to completely revolutionize the way our robotic pan-tilt camera interacts with the physical world.

In Lesson 28, we successfully closed the loop. We got our camera to physically move and track an object using the error signal calculated from our OpenCV bounding box. It worked, but let’s be honest with ourselves: it was clunky. It was a crude, incremental system that moved the camera by exactly one lazy degree at a time, regardless of whether the target was right next to the crosshairs or flying across the room. It was jerky, it hunted back and forth, and it just wasn’t elegant old-school engineering.

Today, we are throwing away that clunky incremental logic and replacing it with something beautiful: Proportional Control.

The Problem with Lazy Incremental Steps

Before we fix our control loop, we need to understand exactly why our previous system struggled. In our last script, we used conditional statements to see if the error was positive or negative, and then adjusted our angles by a fixed step of 1 or -1.

This created two major engineering flaws:

  • Lagging on Large Errors: If you suddenly jerked the object 400 pixels away from the center, the camera would take forever to catch up because it could only step at a constant speed of one degree per loop iteration.

  • Hunting and Jitter on Small Errors: When the object finally got close to the center, the camera would overshoot by a full degree, trip the opposite condition, and step back. It would constantly “hunt” back and forth across the target, buzzing your hardware to pieces.

The Elegance of Proportional Control

In real-world automation, we don’t use rigid, conditional step-programming to move hardware. We use mathematics. We want the camera’s reaction to be completely proportional to the size of the mistake it is trying to correct.

If the object is a massive distance away from the center crosshairs, we want the servo to take a massive, aggressive leap to catch up instantly. As the object gets closer and closer to the center, we want the camera to automatically slow down and gently glide into place. When the error drops to zero, the physical adjustment should naturally drop to zero.

The magic of this approach is that it allows us to completely eliminate the bulky conditional statements and artificial deadbands we wrote last time. The algebra naturally handles the direction and magnitude of the movement.

Breaking Down the Math and Logic

To achieve this fluid motion, we take our raw error signal—the distance in pixels between our frame center and the object center—and apply a scaling factor, known in control theory as Gain.

In this updated system design, we take our error and divide it down. Specifically, we divide the pixel error by 50, and then split that in half by dividing by 2. Mathematically, this means we are scaling our pixel error down by a factor of 100.

  • If your object is 300 pixels off-center, the math calculates an instantaneous adjustment of 3 degrees, quickly snapping the camera toward the target.

  • If the object is only 10 pixels off-center, the adjustment becomes a tiny fraction of a degree (0.1), smoothly stabilizing the camera track.

Precision Tracking with Floating-Point Variables

Because we are dividing our pixel error down by 100, our angular adjustments will almost always be fractional decimals rather than clean integers. If we tried to store these angles as standard integers, our program would truncate those decimals, completely throwing away our precise micro-adjustments and causing the camera to stall out.

To make this system work perfectly, we maintain our accumulation variables as high-precision floating-point numbers. The script constantly adds and subtracts these fractional updates over time behind the scenes. We only cast the final calculated angle to a clean, rounded integer at the absolute last microsecond right as we pass the position command to the physical servo motors.

Visual Tuning and Smooth Performance

You will notice a massive visual upgrade when running this refined loop. To match our new high-precision math, we tighten up our tracking reticle overlay, shrinking our target circle down from a radius of 40 to a crisp 30 pixels. We also change our dynamic bounding box to a bright, vibrant yellow to make our tracking visually pop on screen.

When you fire up this loop and wave your object around, you will see a night-and-day difference compared to last week. The lazy, robotic stutter is completely gone. The pan-tilt mount tracks with an organic, fluid motion, actively accelerating and decelerating to mirror your movements perfectly.

 

AI On the Edge, Python

AI on the Edge LESSON 28: Use Pan Tilt Camera to Track Object of Interest in OpenCV

Hey everyone, Paul McWhorter here from TopTechBoy.com. Welcome back to our channel, where we don’t just write abstract software—we build real, physical, intelligent machines. Go ahead and grab yourself a nice hot cup of coffee or a big glass of iced tea, because today we are closing the loop between the digital world of computer vision and the physical world of robotics.

In our last lesson, we successfully taught OpenCV how to find a specific color, isolate the largest shape, and draw a beautiful green bounding box around it. That was great, but it had a massive limitation: if your object moved off the edge of the frame, it was gone forever. The camera just sat there, blind and helpless.

Today, we change that. We are taking that tracking data from our code and using it to command a physical pan-tilt mechanism powered by two servos on our Fusion Hat. By the time we finish today, your camera will physically turn, tilt, and hunt down your object, keeping it locked dead center in the middle of your video feed.

The Big Leap: Closing the Loop

What we are building today is a foundational concept in automation engineering known as a feedback control loop.

Up until now, your camera was an open-loop observer. It saw things, but it couldn’t react physically. To make an autonomous tracking system, we need to implement a simple pipeline:

  1. Sense: The camera captures the image frame.

  2. Think: OpenCV finds the target object and calculates its position.

  3. Act: The script commands the hardware servos to move the camera mount to correct any positioning errors.

The Mathematics of the Target Error

To make a camera track an object, we have to define what “perfect tracking” looks like to a computer. Perfect tracking means the center of our tracked object is sitting exactly at the center of our video frame.

Because we are running our camera at a crisp resolution of 1280×720, the mathematical center of our universe is fixed. We calculate our frame’s horizontal and vertical centers by dividing our dimensions in half. This gives us a permanent anchor point right in the middle of our grid.

When an object appears on screen, our contour detection gives us its bounding box. We calculate the exact center of that box by taking its starting coordinate and adding half of its width and height. Now we have two sets of coordinates:

  • Where we want the object to be (The Frame Center).

  • Where the object actually is (The Box Center).

The difference between where the object is and where it belongs is called the Error Signal. We calculate an X Error and a Y Error by simply subtracting the frame center from the box center.

Managing Jitter with a Control Deadband

If our object is perfectly centered, our Error is zero. If the object moves to the right, the X Error becomes a positive number. If it moves to the left, it becomes a negative number. The same logic applies vertically to our Y Error.

Now, you might think we should tell the servos to move every single time the Error is anything other than zero. But remember what we learned about camera sensors: pixels dance, light fluctuates, and your calculations will always have a tiny amount of natural mathematical noise. If you try to correct for every single fractional pixel change, your servos will constantly buzz, twitch, and jitter themselves to death.

To fix this, we implement an engineering safety margin called a Deadband. In this lesson, we establish a 40-pixel safety zone around the center of the frame.

  • If the object is within 40 pixels of the center, the error is too small to care about, and we tell the servos to sit perfectly still.

  • The moment the object drifts outside that 40-pixel window, our control logic triggers.

If the X Error is greater than 40, we decrement our pan angle by one degree to turn the camera toward the target, pass that new angle to our servo handler, and pause for a tiny fraction of a second (20 milliseconds) to give the mechanical gears time to physically move. If it’s negative, we increment the angle. We apply the exact same behavioral logic to our tilt servo using the Y Error.

Visualizing the System Matrix

To help us calibrate and troubleshoot this system, we overlay clear visual indicators directly onto our live video feed:

  • The Reticle: We draw a solid blue dot directly at our fixed frame center. This acts as our tracking crosshair.

  • The Target: We draw a large red circle directly over the center of our moving object’s bounding box.

When your system is working properly, you can physically watch the machine think. As you move an object around, the red circle moves away from the blue dot, the error threshold trips, the servos kick in, and the camera moves until the red circle swallows the blue dot once again.

Your Homework Assignment

You guys know the drill: watching me build a tracking rig doesn’t make you an automation engineer. You have to write the logic, feel the hardware move, and tune it yourself.

Here is your homework challenge for Lesson 28: Right now, our tracking logic uses what is called an incremental step controller. No matter how far away the object is from the center, the camera always moves at the exact same speed—one lazy degree at a time. If you move your target slowly, the camera keeps up. If you snap your target quickly across the room, the camera falls behind and loses it because it can’t accelerate.

Your assignment is to upgrade this control loop. Instead of stepping by a hardcoded value of 1, I want you to make the servo adjustment step proportional to the size of the error. If the object is close to the center, it should move gently by a fraction of a degree. If the object takes off like a rocket and creates a massive error signal, the camera should aggressively throw the servos open to catch up instantly.

Get your proportional tracking loops tuned, shoot a video showing your camera tracking a fast-moving object smoothly, upload it to YouTube, and share your link down in the comments below. See you guys in the next lesson!

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 27: Track Objects of Interest in OpenCV Using Contours

AI on the Edge LESSON 27: Track Objects of Interest in OpenCV Using Contours

Hey everyone, Paul McWhorter here from TopTechBoy.com. Welcome back to our channel, where we learn to build real, intelligent systems on edge hardware. Grab yourself a nice hot cup of coffee or a cold glass of iced tea, because today we are taking a massive leap forward in our computer vision journey.

Up until now, we have learned how to configure our cameras, calculate frame rates smoothly, and isolate specific objects based on color using the HSV color space. We built beautiful masks and composite images that show only our target color. But let’s be honest with ourselves: a mask is just a collection of white pixels on a black screen. The computer doesn’t actually know where the object is, how big it is, or how to follow it if it moves.

In this lesson, we are going to fix that. We are going to teach the machine to look at our mask, isolate the single biggest shape of interest, ignore the background noise, and draw a real-time bounding tracking box around it. This is true object tracking.

The Core Concept: What is a Contour?

Think of a contour as a mathematical boundary line. When OpenCV looks at a binary mask (where your target object is white and everything else is black), a contour is the continuous line that traces the outer edge of that white shape.

The beauty of contours is that they turn a chaotic cloud of thousands of isolated pixels into structured, manageable vector shapes. Once OpenCV finds these shapes, it can calculate their physical properties, such as their area, perimeter, and exact center.

The Three Steps to Algorithmic Object Tracking

To turn a raw camera frame into a fully tracked target, our script follows a strict three-part engineering pipeline inside our main execution loop:

1. Extracting Every Boundary

First, we pass our binary mask into OpenCV’s contour detection engine. We configure it to use external retrieval, meaning it will ignore any hollow holes inside the object and only trace the outermost boundary. It returns a list of every single contour it finds in the frame.

2. Hunting for the Largest Target

In the real world, your camera view is never perfectly clean. Even with an excellent HSV color mask, you will get random speckles, reflections, or background noise showing up as tiny white dots on your mask. If we tried to track everything, our program would lose its mind. To solve this, we use a Python maximization function to scan our list of contours and extract the absolute largest one based on its physical area.

3. Setting an Area Noise Floor

Even after finding the largest contour, what happens if your object completely leaves the camera view? The largest remaining “object” might be a tiny, single-pixel spec of static noise on the edge of the screen. To prevent our tracking box from jumping around erratically, we establish a strict structural threshold—a noise floor. If the area of the largest contour isn’t big enough to confidently be our target, we ignore it completely.

Drawing the Bounding Box

Once we have successfully isolated our valid, large contour, we don’t just want to draw a messy, squiggly line around it. We want clean coordinates that an automation system or a robotic pan-tilt kit could actually use to follow the target.

We pass our largest contour into a bounding rectangle function. OpenCV automatically calculates the exact mathematical limits of that shape and returns four precise numbers:

    • X: The horizontal starting pixel coordinate of the object.

    • Y: The vertical starting pixel coordinate of the object.

    • W: The total width of the object in pixels.

    • H: The total height of the object in pixels.

With those four dimensions locked down, we use a standard drawing function to overlay a crisp, green rectangle directly onto our live color camera feed. Now, as you move your object around the room, the box follows it dynamically, tracking its position in real time at high frame rates.

Note you will have to tune the LC and UC parameters for your object of interest, as we showed last week.

 

Making The World a Better Place One High Tech Project at a Time. Enjoy!