Tag Archives: Raspberry Pi 5

AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5

September 28, 2026 admin

Welcome back, everyone! In today’s lesson, AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5, we are building a custom, real-time hand gesture recognition system from scratch. Up until now, we have been using MediaPipe to track keypoints, identify individual hand joints, and extract raw 3D coordinate data. But raw coordinates alone don’t tell the computer what your hand is actually doing. Today, we bridge that gap by teaching our Raspberry Pi 5 how to train on custom hand positions and dynamically classify live hand gestures using pure mathematics and vector geometry—no heavy neural network retraining required.

The core secret to making gesture recognition work reliably across different camera distances is scale-invariant distance normalization. If you simply measure the pixel distance between your fingertips and your wrist, bringing your hand closer to the camera lens will blow out those numbers and break your classifier. To solve this, our script establishes a baseline unit of measurement for every single frame: the distance between the wrist (landmark 0) and the index finger base joint (landmark 5). By dividing all measured finger-tip-to-wrist and tip-to-tip distances by this dynamic scale factor, the resulting feature vector remains identical whether your hand is two feet away or five feet away from the lens.

Once we extract this normalized 15-element feature vector—representing the spatial ratios between all five fingertips relative to the wrist and each other—we construct an interactive training phase directly inside the live video loop. The program prompts you in the terminal for the number of custom gestures you wish to record, along with their labels (such as “Peace”, “Thumbs Up”, or “Fist”). As you present each gesture to your camera feed and press the spacebar, the algorithm captures the instantaneous geometric signature of your hand and stores it in a runtime training dictionary.

During live inference, our classifier calculates the Sum of Absolute Errors (SAE), or Manhattan distance, between the live incoming feature vector and every stored profile in our training dataset. The function loops through all saved gestures to find the candidate with the absolute lowest cumulative error score. To prevent false positives when an untrained or messy hand pose is shown, we compare that lowest error score against a strict threshold value (maxErrorThreshold = 3.5). If the error falls within the allowable boundary, the matched gesture label is instantly displayed on the live OpenCV HUD; otherwise, the system safely defaults to “Unknown”.

We manage our camera feed using high-speed frame capture with Picamera2 running at 60 frames per second at 1280×720 resolution, processing hand detection with MediaPipe Hands, and rendering real-time performance diagnostics—including an exponential moving average FPS counter—directly onto the video output. Go ahead, study the methodology, load the concept onto your Raspberry Pi 5, grab a hot cup of coffee, and let’s get building!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import math

def findDistance(pt1, pt2):
    dX = pt1.x - pt2.x
    dY = pt1.y - pt2.y
    dZ = pt1.z - pt2.z
    dist = math.sqrt(dX*dX + dY*dY + dZ*dZ)
    return dist

def getTipDistances(handLm):
    lm = handLm.landmark
    wrist = lm[0]
    indexBase = lm[5]  
    scaleFactor = findDistance(wrist, indexBase)
    tips = [lm[4],lm[8],lm[12],lm[16],lm[20]]
    tipDistances =[]
    for tip in tips:
        dist = findDistance( wrist, tip) / scaleFactor
        tipDistances.append(dist)
    for startTip in range(len(tips)):
        for endTip in range(startTip+1, len(tips)):
            dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor
            tipDistances.append(dist)
    return tipDistances

def calculateError(tipDist, trainDist):
    totalError = 0
    for idx in range(len(tipDist)):
        diff = tipDist[idx] -trainDist[idx]
        totalError = totalError + abs(diff)
    return totalError

def findGesture(tipLive, trainedData, maxAllowedError):
    bestMatch = "Unknown"
    lowestError = 999999999
    for gestureName, trainedDist in trainedData.items():
        error  = calculateError(tipLive, trainedDist)
        if error < lowestError:
            lowestError = error
            bestMatch = gestureName
    if lowestError > maxAllowedError:
        return "Unknown"
    if lowestError <= maxAllowedError:
        return bestMatch
print("HAND GESTURE TRAINING SETUP")
numGestures = int(input("Enter Number of Gestures to Train On: "))
gestureNames =[]
for idx in range(numGestures):
    name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ")
    gestureNames.append(name)
W=1280
H=720
tStart = time.time()
fps = 15
RES = (W,H)
piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.002
fontColor = (0,0,255)

textFps = (int(W*.01),int(H*.07))
textGesture= (int(W*.01),int(H*.14))
textGesturePrompt = (int(W*.01),int(H*.21))

maxErrorThreshold = 3.5 

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)

trainedData ={}
trainIdx = 0

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    if results.multi_hand_landmarks:
        frameHeight, frameWidth, _ = frame.shape        
        for handLm in results.multi_hand_landmarks:
            # Draw landmark numbers on hand joints
            for idx, landmark in enumerate(handLm.landmark):
                pixelX = int(landmark.x * frameWidth)
                pixelY = int(landmark.y * frameHeight)
                cv2.putText(
                    frame, str(idx), (pixelX, pixelY),
                    cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness
                )
    if trainIdx < len(gestureNames):
        targetName = gestureNames[trainIdx]
        promptText = "show " + targetName + " & Press Space"
        cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness)
        key = cv2.waitKey(1)
        if key == ord(' '):
            if results.multi_hand_landmarks:
                targetHand = results.multi_hand_landmarks[0]
                trainedData[targetName] = getTipDistances(targetHand)
                trainIdx = trainIdx + 1
        if key == ord('q'):
            break
    detectGesture = "Unknown"
    if results.multi_hand_landmarks:
        liveDistances = getTipDistances(results.multi_hand_landmarks[0])
        detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)
    labelText = "Gesture: " + detectGesture
    cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness)
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import math

def findDistance(pt1, pt2):

dX = pt1.x - pt2.x

dY = pt1.y - pt2.y

dZ = pt1.z - pt2.z

dist = math.sqrt(dX*dX + dY*dY + dZ*dZ)

return dist

def getTipDistances(handLm):

lm = handLm.landmark

wrist = lm[0]

indexBase = lm[5]

scaleFactor = findDistance(wrist, indexBase)

tips = [lm[4],lm[8],lm[12],lm[16],lm[20]]

tipDistances =[]

for tip in tips:

dist = findDistance( wrist, tip) / scaleFactor

tipDistances.append(dist)

for startTip in range(len(tips)):

for endTip in range(startTip+1, len(tips)):

dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor

tipDistances.append(dist)

return tipDistances

def calculateError(tipDist, trainDist):

totalError = 0

for idx in range(len(tipDist)):

diff = tipDist[idx] -trainDist[idx]

totalError = totalError + abs(diff)

return totalError

def findGesture(tipLive, trainedData, maxAllowedError):

bestMatch = "Unknown"

lowestError = 999999999

for gestureName, trainedDist in trainedData.items():

error = calculateError(tipLive, trainedDist)

if error < lowestError:

lowestError = error

bestMatch = gestureName

if lowestError > maxAllowedError:

return "Unknown"

if lowestError <= maxAllowedError:

return bestMatch

print("HAND GESTURE TRAINING SETUP")

numGestures = int(input("Enter Number of Gestures to Train On: "))

gestureNames =[]

for idx in range(numGestures):

name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ")

gestureNames.append(name)

W=1280

H=720

tStart = time.time()

fps = 15

RES = (W,H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.002

fontColor = (0,0,255)

textFps = (int(W*.01),int(H*.07))

textGesture= (int(W*.01),int(H*.14))

textGesturePrompt = (int(W*.01),int(H*.21))

maxErrorThreshold = 3.5

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

trainedData ={}

trainIdx = 0

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

frameHeight, frameWidth, _ = frame.shape

for handLm in results.multi_hand_landmarks:

# Draw landmark numbers on hand joints

for idx, landmark in enumerate(handLm.landmark):

pixelX = int(landmark.x * frameWidth)

pixelY = int(landmark.y * frameHeight)

cv2.putText(

frame, str(idx), (pixelX, pixelY),

cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness

)

if trainIdx < len(gestureNames):

targetName = gestureNames[trainIdx]

promptText = "show " + targetName + " & Press Space"

cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness)

key = cv2.waitKey(1)

if key == ord(' '):

if results.multi_hand_landmarks:

targetHand = results.multi_hand_landmarks[0]

trainedData[targetName] = getTipDistances(targetHand)

trainIdx = trainIdx + 1

if key == ord('q'):

break

detectGesture = "Unknown"

if results.multi_hand_landmarks:

liveDistances = getTipDistances(results.multi_hand_landmarks[0])

detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

labelText = "Gesture: " + detectGesture

cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

Hand Recognition on the Raspberry Pi 5 With openCV and Mediapipe

September 25, 2026 admin

This is a base program which allows you to use mediapipe on the Raspberry Pi 5 under openCV to identify each landmark on a hand. This program is important as we can see the index of each landmark. This will serve as the basis for our next program which will allow us to begin to do gesture recogntion.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp

W=1280
H=720
tStart = time.time()
fps = 15
RES = (W,H)
piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.002
fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    if results.multi_hand_landmarks:
        frameHeight, frameWidth, _ = frame.shape        
        for handLm in results.multi_hand_landmarks:
            # Draw landmark numbers on hand joints
            for idx, landmark in enumerate(handLm.landmark):
                pixelX = int(landmark.x * frameWidth)
                pixelY = int(landmark.y * frameHeight)
                cv2.putText(
                    frame, str(idx), (pixelX, pixelY),
                    cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness
                )
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

W=1280

H=720

tStart = time.time()

fps = 15

RES = (W,H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.002

fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

frameHeight, frameWidth, _ = frame.shape

for handLm in results.multi_hand_landmarks:

# Draw landmark numbers on hand joints

for idx, landmark in enumerate(handLm.landmark):

pixelX = int(landmark.x * frameWidth)

pixelY = int(landmark.y * frameHeight)

cv2.putText(

frame, str(idx), (pixelX, pixelY),

cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness

)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 48: Hand Detection in OpenCV and MediaPipe on the Raspberry Pi

September 6, 2026 admin

In Lesson 48 we’re taking a huge step forward — we’re bringing real-time hand detection to the Raspberry Pi using the incredible power of MediaPipe combined with OpenCV and the Pi Camera. This is one of those projects that feels like pure magic when you see it working: the camera picks up your hands instantly, tracks all 21 landmarks on each hand, draws beautiful connections in real time, and does it all right on the edge with no cloud required.

You’ll learn how to set up MediaPipe’s Hands solution for reliable multi-hand tracking, how to efficiently process frames from the Picamera2 library, flip and convert images for proper display, and draw professional-looking landmarks and connections with custom styling. We also keep a smooth FPS counter running so you can see exactly how well your Pi is performing.

This lesson is exciting because hand tracking is the foundation for so many advanced gesture-control projects — think touchless interfaces, sign language recognition, robotic control, virtual instruments, and interactive art installations. Once you have solid hand detection running smoothly on your Raspberry Pi, the creative possibilities are almost endless.

So grab your Raspberry Pi, fire up the camera, and let’s get those hands dancing on the screen! As always, the full code is waiting for you below. Watch the video, follow along, and then start experimenting — I can’t wait to see what you build with this!

Let’s make something awesome!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================

import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)
mpDraw = mp.solutions.drawing_utils

handStyle = mpDraw.DrawingSpec(
    color = (100, 0, 255),
    thickness = 6,
    circle_radius = 8)
connectionStyle = mpDraw.DrawingSpec(
    color = (0,255,0),
    thickness = 5)

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    if results.multi_hand_landmarks:
        for handLM in results.multi_hand_landmarks:
            mpDraw.draw_landmarks(
                frame,
                handLM,
                mp.solutions.hands.HAND_CONNECTIONS,
                handStyle,
                connectionStyle)
        
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

mpDraw = mp.solutions.drawing_utils

handStyle = mpDraw.DrawingSpec(

color = (100, 0, 255),

thickness = 6,

circle_radius = 8)

connectionStyle = mpDraw.DrawingSpec(

color = (0,255,0),

thickness = 5)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

for handLM in results.multi_hand_landmarks:

mpDraw.draw_landmarks(

frame,

handLM,

mp.solutions.hands.HAND_CONNECTIONS,

handStyle,

connectionStyle)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 33: Tracking Faces with Pan Tilt Camera in OpenCV on Pi 5

July 16, 2026 admin

Hey guys, Paul McWhorter here from toptechboy.com. In today’s lesson, we are taking our “AI on the Edge” skills to the next level. We aren’t just detecting faces anymore; we are going to make our camera system react to them.

We are integrating our computer vision logic with physical hardware to create a pan-tilt tracking system. We’ll use the Raspberry Pi 5 to run high-speed inference, detect a face, calculate exactly how far that face has drifted from the center of our frame, and then command our servos to follow it in real-time. It is one thing to see a box draw around a face; it is a completely different level of “cool” when the camera actually turns to look at you.

The Engineering Concept: The Error Loop

In robotics, this is a classic control problem. We have a Target (the center of the face) and a Setpoint (the center of our camera frame). The difference between these two points is our Error.

xError: How far left or right the face is from the center.
yError: How far up or down the face is from the center.

By taking that error and dividing it by a “gain” constant (in our case, 50/2), we can smoothly adjust the servo angles. If we don’t divide by a constant, the camera will snap aggressively to the target or overshoot it. This simple division creates a “proportional” response that keeps our tracking smooth and precise.

What to Focus On

Make sure you have your picamera2 and fusion_hat libraries updated and configured before you dive in. The key to this lesson isn’t just getting the servos to move—it’s understanding how to bridge the gap between the coordinates returned by OpenCV and the angle coordinates required by your servo library.

Pay close attention to how we calculate the center of the frame and the center of the detected face box. Once you understand that math, you can use this same logic to track anything: faces, colored objects, or even specific shapes!

Homework Assignment: Show Your Work!

Alright guys, no excuses! If you want to truly master this hardware, you cannot just sit there and watch me do it—you have to write the code and run it yourself.

Your Task: Get this tracking system working. Once you have it tracking a face, I want you to experiment with that “Gain” factor (the 50/2 part). Try increasing it and decreasing it. What happens to the tracking quality? Is it smoother? Does it jitter?

Record a video of your camera following your face around the room, upload it to YouTube, and link back to this video at the top of your page. Post a link to your homework in the comments section below so I can see you running your code like a boss.

The code we developed in the video is available below.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
from fusion_hat.servo import Servo
panPin = 2
tiltPin = 3
panServo = Servo(panPin)
tiltServo = Servo(tiltPin)
panAngle = -10
tiltAngle =-20
panServo.angle(panAngle)
tiltServo.angle(tiltAngle)
piCam = Picamera2()
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

faceFinder = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')
eyeFinder = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_eye.xml")

track = False
xFrameC = int((W-1)/2)
yFrameC = int((H-1)/2)
xBoxC = xFrameC
yBoxC = yFrameC

def mouseAction(event, x, y, flags, param):
    global frame, track
    if event == 0:
        xPos = x
        yPos = y
    if frame is not None:
        frameHSV = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
        Hue, Sat, Val = frameHSV[y,x]
        if Hue == 60:
            track = True
        if Hue == 0:
            track = False
cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)
cv2.setMouseCallback('Camera',mouseAction)

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    
    cv2.circle(frame,(int(.9*W),int(.1*H)),20,(0,255,0),-1)
    cv2.circle(frame, (int(.95*W),int(.1*H)),20,(0,0,255),-1)

    gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)
    faces = faceFinder.detectMultiScale(gray,scaleFactor=1.1, minNeighbors =5, minSize=(100,100))
    for (x,y,w,h) in faces:
        cv2.rectangle(frame, (x,y),(x+w,y+h),(0,255,255),3)
        
        xBoxC = x + int(w/2)
        yBoxC = y + int(h/2)
        xError = xBoxC - xFrameC
        yError = yBoxC - yFrameC
        
        eyes =eyeFinder.detectMultiScale(gray[y:y+h,x:x+w],scaleFactor=1.1,minNeighbors=5,minSize=(15,15))
        for (i,j,w,h) in eyes:
            cv2.rectangle(frame,(x+i,y+j),(x+i+w,y+j+h),(255,0,0),3)
    
    if track == True and len(faces)>0:
        panAngle = panAngle - xError/50/2
        panServo.angle(int(panAngle))
        #time.sleep(.02)
        tiltAngle = tiltAngle + yError/50/2
        tiltServo.angle(int(tiltAngle))
        #time.sleep(.02)
                        
    
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

from fusion_hat.servo import Servo

panPin = 2

tiltPin = 3

panServo = Servo(panPin)

tiltServo = Servo(tiltPin)

panAngle = -10

tiltAngle =-20

panServo.angle(panAngle)

tiltServo.angle(tiltAngle)

piCam = Picamera2()

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

faceFinder = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')

eyeFinder = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_eye.xml")

track = False

xFrameC = int((W-1)/2)

yFrameC = int((H-1)/2)

xBoxC = xFrameC

yBoxC = yFrameC

def mouseAction(event, x, y, flags, param):

global frame, track

if event == 0:

xPos = x

yPos = y

if frame is not None:

frameHSV = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

Hue, Sat, Val = frameHSV[y,x]

if Hue == 60:

track = True

if Hue == 0:

track = False

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

cv2.setMouseCallback('Camera',mouseAction)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

cv2.circle(frame,(int(.9*W),int(.1*H)),20,(0,255,0),-1)

cv2.circle(frame, (int(.95*W),int(.1*H)),20,(0,0,255),-1)

gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)

faces = faceFinder.detectMultiScale(gray,scaleFactor=1.1, minNeighbors =5, minSize=(100,100))

for (x,y,w,h) in faces:

cv2.rectangle(frame, (x,y),(x+w,y+h),(0,255,255),3)

xBoxC = x + int(w/2)

yBoxC = y + int(h/2)

xError = xBoxC - xFrameC

yError = yBoxC - yFrameC

eyes =eyeFinder.detectMultiScale(gray[y:y+h,x:x+w],scaleFactor=1.1,minNeighbors=5,minSize=(15,15))

for (i,j,w,h) in eyes:

cv2.rectangle(frame,(x+i,y+j),(x+i+w,y+j+h),(255,0,0),3)

if track == True and len(faces)>0:

panAngle = panAngle - xError/50/2

panServo.angle(int(panAngle))

#time.sleep(.02)

tiltAngle = tiltAngle + yError/50/2

tiltServo.angle(int(tiltAngle))

#time.sleep(.02)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 31: Facial Recognition in OpenCV Using Haarcascades

July 8, 2026 admin

Hey everyone, this is Paul McWhorter from toptechboy.com. Welcome back to our AI on the Edge tutorial series. If you’ve made it this short distance through the course, you are doing fantastic.

Today, we are stepping away from simply pulling a clean, high-frame-rate video stream off our hardware, and we are finally going to start doing some real Computer Vision. We are going to teach our machine how to look at an image, process it, and actually locate human faces in real time.

Go ahead and grab yourself a hot cup of coffee. Turn off your cell phone, close your other browser tabs, and let’s get ready to write some serious engineering code.

The Core Concept: What is a Haar Cascade?

Up until now, we’ve been focused on the plumbing—getting the camera configured, setting up the dimensions, and flipping the frames so they look right on our screens. Today, we introduce the Haar Cascade Classifier.

Think of a Haar Cascade as an incredibly smart, pre-trained statistical filter. Decades ago, researchers realized that human faces share universal geometric patterns of light and dark. For example:

The bridge of your nose is almost always brighter than the shadows on either side of it.
Your eye sockets are consistently darker than your forehead or your cheekbones.

OpenCV comes pre-packaged with these mathematical descriptions of a face. The algorithm works by taking a tiny “sliding window” and dragging it across your video frame pixel by pixel, looking for those specific arrangements of light and dark gradients. When it finds a cluster that matches the math, it flags it as a face.

Why Grayscale Matters in Machine Learning

If you look closely at our pipeline logic for today, the absolute first thing we do after capturing a raw frame from our camera is strip away all the color and convert the image to grayscale.

Why do we do this? Color is a computational luxury we cannot afford when doing real-time edge processing. To a computer, a color image consists of three separate channels: Red, Green, and Blue. That means for a standard resolution, the processor has to crunch three times the data.

Because Haar Cascades only care about the contrast—the relationship between light and dark areas—the actual color of your skin, your hair, or your shirt is completely irrelevant. By dropping the frame down to a single grayscale intensity channel, we cut our processor’s workload by 66% instantly. This is how we keep our edge hardware running lightning-fast without lagging the video feed.

Tuning the Detection Engine

When we tell OpenCV to look for faces using the detectMultiScale function, we pass three critical parameters that you need to master. If you don’t tune these right, your program will either miss faces entirely or start thinking your coffee mug or a pattern on the wall is a human being.

scaleFactor=1.1: A single Haar Cascade model is trained at a fixed size. But a face could be right next to the lens (huge) or all the way across the room (tiny). The scaleFactor tells OpenCV to shrink the image by 10% on each pass, creating a “layer cake” of images at different scales so the sliding window can catch faces of any size.
minNeighbors=5: As the sliding window moves, it might find dozens of potential matches around your eyes and nose. This parameter states that a face must be detected in at least 5 overlapping bounding boxes before the system officially declares, “Yes, that is a human face.” Raising this number reduces false positives but makes the system less sensitive.
minSize=(60,60): This tells the algorithm to completely ignore any detected objects smaller than a 60×60 pixel block. This prevents the system from wasting CPU cycles trying to analyze tiny bits of background noise in the distance.

The Secret to a Smooth FPS Counter

You will notice a very specific mathematical formula used to calculate our Frames Per Second (FPS) in this lesson. If you simply calculate $1 / \Delta T$ every single frame, your FPS display will jitter wildly on screen, flashing unreadable numbers back and forth because individual frames might take a millisecond more or less to process.

To fix this, we implement a digital Low-Pass Filter.

Every loop, we take 95% of our previous FPS value and add only 5% of our current instantaneous speed measurement. This creates a beautiful, smooth running average that responds immediately to system slowdowns but remains steady and completely legible on the screen. It is an elegant engineering solution to a common UI headache.

Homework Assignment!

You didn’t think you were going to get out of here without homework, did you? No shortcuts in this classroom!

Now that your program can successfully locate faces and draw a bounding box around them, it’s time to take it to the next level. Your assignment is to modify this program to isolate and track only the single largest face in the frame.

If multiple people walk into the camera’s view, your script must evaluate the dimensions of the returned bounding boxes, determine which person is closest to the lens (the largest box), and draw a bright green rectangle around only that leader face, while ignoring everyone else.

Here is the code developed in today’s lesson.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
piCam = Picamera2()
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

faceFinder = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)

    gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)
    faces = faceFinder.detectMultiScale(gray,scaleFactor=1.1, minNeighbors =5, minSize=(60,60))
    for (x,y,w,h) in faces:
        cv2.rectangle(frame, (x,y),(x+w,y+h),(0,255,255),3)
    
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

piCam = Picamera2()

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

faceFinder = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

gray = cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)

faces = faceFinder.detectMultiScale(gray,scaleFactor=1.1, minNeighbors =5, minSize=(60,60))

for (x,y,w,h) in faces:

cv2.rectangle(frame, (x,y),(x+w,y+h),(0,255,255),3)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

Technology Tutorials

Tag Archives: Raspberry Pi 5

AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5

Hand Recognition on the Raspberry Pi 5 With openCV and Mediapipe

AI on the Edge LESSON 48: Hand Detection in OpenCV and MediaPipe on the Raspberry Pi

AI on the Edge LESSON 33: Tracking Faces with Pan Tilt Camera in OpenCV on Pi 5

The Engineering Concept: The Error Loop

What to Focus On

Homework Assignment: Show Your Work!

AI on the Edge LESSON 31: Facial Recognition in OpenCV Using Haarcascades

The Core Concept: What is a Haar Cascade?

Why Grayscale Matters in Machine Learning

Tuning the Detection Engine

The Secret to a Smooth FPS Counter

Homework Assignment!

Making The World a Better Place One High Tech Project at a Time. Enjoy!