Tag Archives: Fusion AI Lab Kit

Welcome back, everyone! This is Paul McWhorter from TopTechBoy.com, and welcome to Lesson 55 in our AI on the Edge tutorial series!

If you’ve been following along, grab yourself a nice, hot cup of black coffee, fire up your Raspberry Pi 5, and let’s get right into it!

What We Are Doing in Lesson 55

Up to this point in the series, we have spent a lot of time working with land-marking models—tracking human hands, pose points, and facial features. But today, we step into full-blown computer vision object detection!

In this lesson, we learn how to implement MediaPipe’s lightweight, standalone object detection task API (mediapipe.tasks.python.vision.ObjectDetector) running on the Raspberry Pi 5. Using the pre-trained COCO dataset model (efficientdet_lite0.tflite), our Pi 5 can instantly identify up to 80 different real-world objects—from people, cups, and bottles to chairs, laptops, and cell phones—and draw real-time bounding boxes around them on our video feed.

Key Takeaways & Concepts Covered

Setting Up the MediaPipe Object Detector Task API:

Unlike older mp.solutions wrappers, the modern MediaPipe Tasks API requires configuring BaseOptions with the explicit local path to your .tflite model asset (modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite").
Configuring Detector Parameters:
- score_threshold (0.35): Sets the minimum confidence score required to accept a detected object. Lowering this value captures weaker detections but risks false positives; raising it ensures high-confidence target lock.
- max_results (3): Limits the maximum number of bounding boxes predicted per frame to keep our processing loop lean and fast.
Decoding Detection Bounding Boxes:

MediaPipe returns detected objects with bounding box geometry relative to the frame origin (origin_x, origin_y, width, height). We parse these dimensions directly into OpenCV integers (x, y, w, h) to draw bounding boxes with cv2.rectangle().
Category Labeling & Confidence Rendering:

We pull category names (category_name) and classification scores (category.score) from detection.categories[0], format the confidence as a percentage, and render clean text labels anchored directly above each target box.
Optimal Frame Alignment & FPS Monitoring:

We capture 60 FPS video directly from picamera2 in RGB888 format, flip it properly to match our physical camera mounting, and track real-time frames-per-second using our low-pass smoothing formula.

What You Will Need

To follow along with this lesson, ensure you have your standard gear ready from the kit:

Raspberry Pi 5 (with Active Cooler/Heat Sink)
Raspberry Pi Camera Module 3 (or compatible Camera Module driven via picamera2)
SunFounder Fusion AI Lab Kit (with pre-downloaded MediaPipe .tflite model weights)

Homework Assignment

Alright, class, you know I never let you off the hook without some homework!

Your assignment for this lesson is to take this foundational object detection script and upgrade its intelligence:

Target Filter: Modify the loop so that it only draws bounding boxes around a specific target object of your choice (for instance, only detect "person" or only detect "bottle"), ignoring all other detected COCO classes.
Target Tracking Dot: Calculate the exact center pixel coordinate $(x + \frac{w}{2}, y + \frac{h}{2})$ of your targeted object, and draw a solid red target circle (cv2.circle) right on the object’s center point.
Dynamic On-Screen Target Status: If your target object is visible in the frame, display a green text banner in the upper corner reading "TARGET ACQUIRED". If it leaves the frame, change the status banner to a red "SEARCHING...".

Give it a shot, take your time, and write your solution from scratch without copying and pasting! Drop a comment on the YouTube video to let me know if you got it working or if you ran into any snags.

Until next time, get your coffee, keep coding, and I’ll see you guys in Lesson 56!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision

piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
topBar = 65
windowPadding = 0
cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)
cv2.resizeWindow('Camera', W, H)
tSettle = .1
time.sleep(tSettle)
cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"
scoreThreshold = .35
maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)
options = vision.ObjectDetectorOptions(base_options = baseOptions,
                                       score_threshold=scoreThreshold,
                                       max_results=maxObj)
myDetector = vision.ObjectDetector.create_from_options(options)

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)
    myDetections = myDetector.detect(myImage)
    #print(myDetections.detections)
    for detection in myDetections.detections:
        box = detection.bounding_box
        x = box.origin_x
        y = box.origin_y
        w = box.width
        h = box.height
        cv2.rectangle(frame, (x,y), (x+w,y+h), (255,0,0), fontThickness)
    if detection.categories:
        category = detection.categories[0]
        myLabel = category.category_name
        labelText = myLabel+" "+ str(round(category.score*100))+"%"
        cv2.putText(frame,labelText, (x, max(y-10,25)),fontFace,
                                      fontScale*.8,
                                      (0,255,0),
                                      2)
                                      
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
myDetector.close()
cv2.destroyAllWindows()
print('Program Terminated')

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

from mediapipe.tasks import python

from mediapipe.tasks.python import vision

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

topBar = 65

windowPadding = 0

cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)

cv2.resizeWindow('Camera', W, H)

tSettle = .1

time.sleep(tSettle)

cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"

scoreThreshold = .35

maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)

options = vision.ObjectDetectorOptions(base_options = baseOptions,

score_threshold=scoreThreshold,

max_results=maxObj)

myDetector = vision.ObjectDetector.create_from_options(options)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)

myDetections = myDetector.detect(myImage)

#print(myDetections.detections)

for detection in myDetections.detections:

box = detection.bounding_box

x = box.origin_x

y = box.origin_y

w = box.width

h = box.height

cv2.rectangle(frame, (x,y), (x+w,y+h), (255,0,0), fontThickness)

if detection.categories:

category = detection.categories[0]

myLabel = category.category_name

labelText = myLabel+" "+ str(round(category.score*100))+"%"

cv2.putText(frame,labelText, (x, max(y-10,25)),fontFace,

fontScale*.8,

(0,255,0),

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

myDetector.close()

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 53: Gesture Recognition on the Pi 5 with Training File

October 4, 2026 admin

Welcome back, everyone! In this lesson, we are taking our custom MediaPipe hand gesture recognition system to the next level. Up until now, every time you stopped your Python script, all of your hard-earned gesture training data vanished into thin air. If you wanted to run your project again, you had to re-train every single hand sign from scratch.

That ends today. We are going to make our AI remember what it learned by saving our gesture feature vectors to a local file using Python’s pickle library.

What You Will Learn in This Lesson

Data Persistence: How to serialize and save Python dictionaries containing landmark distance vectors directly to disk (gesture_data.pkl).
Automated Setup: How to check for existing training files at boot so your program automatically skips the initial training prompt if data already exists.
Dynamic Training Management: How to interactively train new gestures on the fly ('t' key) or delete outdated gestures ('d' key) during runtime while automatically updating the saved file.
Robust Error Handling: Ensuring index tracking (trainIdx) safely syncs with live gesture arrays so pressing the spacebar never crashes your video feed.

Key Concepts Explained

1. Pickling Data Objects In MediaPipe gesture recognition, a trained gesture isn’t a complex neural network weight file; it’s simply a dictionary mapping a gesture name (string) to a list of normalized keypoint ratios (floating-point numbers). Python’s pickle module allows us to convert this dictionary into a byte stream and write it directly to a file (.pkl). When the program launches, pickle.load() reconstructs the dictionary in milliseconds.

2. Managing Execution Flow on Startup When gesture_data.pkl exists, our program automatically populates gestureNames with the keys from trainedData and sets trainIdx past the end of the array. This keeps the camera pipeline running in Inference Mode immediately. If no file exists, the script seamlessly drops into Training Setup Mode to prompt the user for initial inputs.

This is the code we develop in this video lesson:

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import math
import os
import pickle

DATA_FILE = "gesture_data.pkl"

def saveTrainedData(filename, data):
    with open(filename, 'wb') as f:
        pickle.dump(data,f)
    print("[INFO] Saved Gesture Data to "+str(filename))
def loadTrainedData(filename):
    if os.path.exists(filename):
        with open(filename, 'rb') as f:
            data = pickle.load(f)
            print("[INFO] LOADED "+str(len(data))+ " gestures from " + str(filename))
            return data
    if not os.path.exists(filename):
        return {}

def findDistance(pt1, pt2):
    dX = pt1.x - pt2.x
    dY = pt1.y - pt2.y
    dZ = pt1.z - pt2.z
    dist = math.sqrt(dX*dX + dY*dY + dZ*dZ)
    return dist

def getTipDistances(handLm):
    lm = handLm.landmark
    wrist = lm[0]
    indexBase = lm[5]  
    scaleFactor = findDistance(wrist, indexBase)
    tips = [lm[4],lm[8],lm[12],lm[16],lm[20]]
    tipDistances =[]
    for tip in tips:
        dist = findDistance( wrist, tip) / scaleFactor
        tipDistances.append(dist)
    for startTip in range(len(tips)):
        for endTip in range(startTip+1, len(tips)):
            dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor
            tipDistances.append(dist)
    return tipDistances

def calculateError(tipDist, trainDist):
    totalError = 0
    for idx in range(len(tipDist)):
        diff = tipDist[idx] -trainDist[idx]
        totalError = totalError + abs(diff)
    return totalError

def findGesture(tipLive, trainedData, maxAllowedError):
    bestMatch = "Unknown"
    lowestError = 999999999
    for gestureName, trainedDist in trainedData.items():
        error  = calculateError(tipLive, trainedDist)
        if error < lowestError:
            lowestError = error
            bestMatch = gestureName
    if lowestError > maxAllowedError:
        return "Unknown"
    if lowestError <= maxAllowedError:
        return bestMatch
trainedData = loadTrainedData(DATA_FILE)
gestureNames =[]
if not trainedData:
    print("HAND GESTURE TRAINING SETUP")
    numGestures = int(input("Enter Number of Gestures to Train On: "))
    gestureNames =[]
    for idx in range(numGestures):
        name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ")
        gestureNames.append(name)
W=1280
H=720
tStart = time.time()
fps = 15
RES = (W,H)
piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.002
fontColor = (0,0,255)

textFps = (int(W*.01),int(H*.07))
textGesture= (int(W*.01),int(H*.14))
textGesturePrompt = (int(W*.01),int(H*.21))

maxErrorThreshold = 3.5 

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)
trainIdx = 0

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    if results.multi_hand_landmarks:
        frameHeight, frameWidth, _ = frame.shape        
        for handLm in results.multi_hand_landmarks:
            # Draw landmark numbers on hand joints
            for idx, landmark in enumerate(handLm.landmark):
                pixelX = int(landmark.x * frameWidth)
                pixelY = int(landmark.y * frameHeight)
                cv2.putText(
                    frame, str(idx), (pixelX, pixelY),
                    cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness
                )
    if trainIdx < len(gestureNames):
        targetName = gestureNames[trainIdx]
        promptText = "show " + targetName + " & Press Space"
        cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness)
    detectGesture = "Unknown"
    if results.multi_hand_landmarks:
        liveDistances = getTipDistances(results.multi_hand_landmarks[0])
        detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)
    labelText = "Gesture: " + detectGesture
    cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness)
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera",0,60)
    key = cv2.waitKey(1)
    if key == ord('t'):
        newGestureName = input("Enter Name for New Gesture: ")
        gestureNames.append(newGestureName)
        saveTrainedData(DATA_FILE, trainedData)
    if key == ord('d'):
        delName = input("Enter Name of Gesture to Remove: ")
        if delName not in trainedData:
            print("[WARNING] Gesture "+delName+ " Not Found in Training Data.")
        if delName in trainedData:
            del trainedData[delName]
            saveTrainedData(DATA_FILE,trainedData)
            print("[INFO] Removed "+delName+"from trained data.")
            if delName in gestureNames:
                gestureNames.remove(delName)
            trainIdx=min(trainIdx,len(gestureNames))
    if key == ord(' '):
        if results.multi_hand_landmarks:
            targetHand = results.multi_hand_landmarks[0]
            trainedData[targetName] = getTipDistances(targetHand)
            trainIdx = trainIdx + 1
    if key == ord('q'):
        break
cv2.destroyAllWindows()
saveTrainedData(DATA_FILE, trainedData)
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import math

import os

import pickle

DATA_FILE = "gesture_data.pkl"

def saveTrainedData(filename, data):

with open(filename, 'wb') as f:

pickle.dump(data,f)

print("[INFO] Saved Gesture Data to "+str(filename))

def loadTrainedData(filename):

if os.path.exists(filename):

with open(filename, 'rb') as f:

data = pickle.load(f)

print("[INFO] LOADED "+str(len(data))+ " gestures from " + str(filename))

return data

if not os.path.exists(filename):

return {}

def findDistance(pt1, pt2):

dX = pt1.x - pt2.x

dY = pt1.y - pt2.y

dZ = pt1.z - pt2.z

dist = math.sqrt(dX*dX + dY*dY + dZ*dZ)

return dist

def getTipDistances(handLm):

lm = handLm.landmark

wrist = lm[0]

indexBase = lm[5]

scaleFactor = findDistance(wrist, indexBase)

tips = [lm[4],lm[8],lm[12],lm[16],lm[20]]

tipDistances =[]

for tip in tips:

dist = findDistance( wrist, tip) / scaleFactor

tipDistances.append(dist)

for startTip in range(len(tips)):

for endTip in range(startTip+1, len(tips)):

dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor

tipDistances.append(dist)

return tipDistances

def calculateError(tipDist, trainDist):

totalError = 0

for idx in range(len(tipDist)):

diff = tipDist[idx] -trainDist[idx]

totalError = totalError + abs(diff)

return totalError

def findGesture(tipLive, trainedData, maxAllowedError):

bestMatch = "Unknown"

lowestError = 999999999

for gestureName, trainedDist in trainedData.items():

error = calculateError(tipLive, trainedDist)

if error < lowestError:

lowestError = error

bestMatch = gestureName

if lowestError > maxAllowedError:

return "Unknown"

if lowestError <= maxAllowedError:

return bestMatch

trainedData = loadTrainedData(DATA_FILE)

gestureNames =[]

if not trainedData:

print("HAND GESTURE TRAINING SETUP")

numGestures = int(input("Enter Number of Gestures to Train On: "))

gestureNames =[]

for idx in range(numGestures):

name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ")

gestureNames.append(name)

W=1280

H=720

tStart = time.time()

fps = 15

RES = (W,H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.002

fontColor = (0,0,255)

textFps = (int(W*.01),int(H*.07))

textGesture= (int(W*.01),int(H*.14))

textGesturePrompt = (int(W*.01),int(H*.21))

maxErrorThreshold = 3.5

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

trainIdx = 0

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

frameHeight, frameWidth, _ = frame.shape

for handLm in results.multi_hand_landmarks:

# Draw landmark numbers on hand joints

for idx, landmark in enumerate(handLm.landmark):

pixelX = int(landmark.x * frameWidth)

pixelY = int(landmark.y * frameHeight)

cv2.putText(

frame, str(idx), (pixelX, pixelY),

cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness

)

if trainIdx < len(gestureNames):

targetName = gestureNames[trainIdx]

promptText = "show " + targetName + " & Press Space"

cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness)

detectGesture = "Unknown"

if results.multi_hand_landmarks:

liveDistances = getTipDistances(results.multi_hand_landmarks[0])

detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

labelText = "Gesture: " + detectGesture

cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera",0,60)

key = cv2.waitKey(1)

if key == ord('t'):

newGestureName = input("Enter Name for New Gesture: ")

gestureNames.append(newGestureName)

saveTrainedData(DATA_FILE, trainedData)

if key == ord('d'):

delName = input("Enter Name of Gesture to Remove: ")

if delName not in trainedData:

print("[WARNING] Gesture "+delName+ " Not Found in Training Data.")

if delName in trainedData:

del trainedData[delName]

saveTrainedData(DATA_FILE,trainedData)

print("[INFO] Removed "+delName+"from trained data.")

if delName in gestureNames:

gestureNames.remove(delName)

trainIdx=min(trainIdx,len(gestureNames))

if key == ord(' '):

if results.multi_hand_landmarks:

targetHand = results.multi_hand_landmarks[0]

trainedData[targetName] = getTipDistances(targetHand)

trainIdx = trainIdx + 1

if key == ord('q'):

break

cv2.destroyAllWindows()

saveTrainedData(DATA_FILE, trainedData)

print('Program Terminated')

Homework Assignment

Now that your Pi 5 can remember hand gestures across reboots, it’s time to bridge software recognition with real-world hardware control!

The Challenge: Integrate an SPI NeoPixel Ring (WS2812B/SK6812) with your gesture recognition code.

Map distinct hand signs to dynamic LED patterns (for example: Fist = Red Pulse, Open Palm = Bright White Beacon, Peace Sign = Dual Spinner, OK Sign = Rainbow Wheel).
Make sure your animation updates are smooth and non-blocking so your OpenCV camera frame rate doesn’t drop!

We want to see who in the community can build the most impressive physical light demonstration controlled entirely by hand gestures. Share your video results and code modifications in the comments below!

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5

September 28, 2026 admin

Welcome back, everyone! In today’s lesson, AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5, we are building a custom, real-time hand gesture recognition system from scratch. Up until now, we have been using MediaPipe to track keypoints, identify individual hand joints, and extract raw 3D coordinate data. But raw coordinates alone don’t tell the computer what your hand is actually doing. Today, we bridge that gap by teaching our Raspberry Pi 5 how to train on custom hand positions and dynamically classify live hand gestures using pure mathematics and vector geometry—no heavy neural network retraining required.

The core secret to making gesture recognition work reliably across different camera distances is scale-invariant distance normalization. If you simply measure the pixel distance between your fingertips and your wrist, bringing your hand closer to the camera lens will blow out those numbers and break your classifier. To solve this, our script establishes a baseline unit of measurement for every single frame: the distance between the wrist (landmark 0) and the index finger base joint (landmark 5). By dividing all measured finger-tip-to-wrist and tip-to-tip distances by this dynamic scale factor, the resulting feature vector remains identical whether your hand is two feet away or five feet away from the lens.

Once we extract this normalized 15-element feature vector—representing the spatial ratios between all five fingertips relative to the wrist and each other—we construct an interactive training phase directly inside the live video loop. The program prompts you in the terminal for the number of custom gestures you wish to record, along with their labels (such as “Peace”, “Thumbs Up”, or “Fist”). As you present each gesture to your camera feed and press the spacebar, the algorithm captures the instantaneous geometric signature of your hand and stores it in a runtime training dictionary.

During live inference, our classifier calculates the Sum of Absolute Errors (SAE), or Manhattan distance, between the live incoming feature vector and every stored profile in our training dataset. The function loops through all saved gestures to find the candidate with the absolute lowest cumulative error score. To prevent false positives when an untrained or messy hand pose is shown, we compare that lowest error score against a strict threshold value (maxErrorThreshold = 3.5). If the error falls within the allowable boundary, the matched gesture label is instantly displayed on the live OpenCV HUD; otherwise, the system safely defaults to “Unknown”.

We manage our camera feed using high-speed frame capture with Picamera2 running at 60 frames per second at 1280×720 resolution, processing hand detection with MediaPipe Hands, and rendering real-time performance diagnostics—including an exponential moving average FPS counter—directly onto the video output. Go ahead, study the methodology, load the concept onto your Raspberry Pi 5, grab a hot cup of coffee, and let’s get building!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import math

def findDistance(pt1, pt2):
    dX = pt1.x - pt2.x
    dY = pt1.y - pt2.y
    dZ = pt1.z - pt2.z
    dist = math.sqrt(dX*dX + dY*dY + dZ*dZ)
    return dist

def getTipDistances(handLm):
    lm = handLm.landmark
    wrist = lm[0]
    indexBase = lm[5]  
    scaleFactor = findDistance(wrist, indexBase)
    tips = [lm[4],lm[8],lm[12],lm[16],lm[20]]
    tipDistances =[]
    for tip in tips:
        dist = findDistance( wrist, tip) / scaleFactor
        tipDistances.append(dist)
    for startTip in range(len(tips)):
        for endTip in range(startTip+1, len(tips)):
            dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor
            tipDistances.append(dist)
    return tipDistances

def calculateError(tipDist, trainDist):
    totalError = 0
    for idx in range(len(tipDist)):
        diff = tipDist[idx] -trainDist[idx]
        totalError = totalError + abs(diff)
    return totalError

def findGesture(tipLive, trainedData, maxAllowedError):
    bestMatch = "Unknown"
    lowestError = 999999999
    for gestureName, trainedDist in trainedData.items():
        error  = calculateError(tipLive, trainedDist)
        if error < lowestError:
            lowestError = error
            bestMatch = gestureName
    if lowestError > maxAllowedError:
        return "Unknown"
    if lowestError <= maxAllowedError:
        return bestMatch
print("HAND GESTURE TRAINING SETUP")
numGestures = int(input("Enter Number of Gestures to Train On: "))
gestureNames =[]
for idx in range(numGestures):
    name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ")
    gestureNames.append(name)
W=1280
H=720
tStart = time.time()
fps = 15
RES = (W,H)
piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.002
fontColor = (0,0,255)

textFps = (int(W*.01),int(H*.07))
textGesture= (int(W*.01),int(H*.14))
textGesturePrompt = (int(W*.01),int(H*.21))

maxErrorThreshold = 3.5 

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)

trainedData ={}
trainIdx = 0

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    if results.multi_hand_landmarks:
        frameHeight, frameWidth, _ = frame.shape        
        for handLm in results.multi_hand_landmarks:
            # Draw landmark numbers on hand joints
            for idx, landmark in enumerate(handLm.landmark):
                pixelX = int(landmark.x * frameWidth)
                pixelY = int(landmark.y * frameHeight)
                cv2.putText(
                    frame, str(idx), (pixelX, pixelY),
                    cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness
                )
    if trainIdx < len(gestureNames):
        targetName = gestureNames[trainIdx]
        promptText = "show " + targetName + " & Press Space"
        cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness)
        key = cv2.waitKey(1)
        if key == ord(' '):
            if results.multi_hand_landmarks:
                targetHand = results.multi_hand_landmarks[0]
                trainedData[targetName] = getTipDistances(targetHand)
                trainIdx = trainIdx + 1
        if key == ord('q'):
            break
    detectGesture = "Unknown"
    if results.multi_hand_landmarks:
        liveDistances = getTipDistances(results.multi_hand_landmarks[0])
        detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)
    labelText = "Gesture: " + detectGesture
    cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness)
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import math

def findDistance(pt1, pt2):

dX = pt1.x - pt2.x

dY = pt1.y - pt2.y

dZ = pt1.z - pt2.z

dist = math.sqrt(dX*dX + dY*dY + dZ*dZ)

return dist

def getTipDistances(handLm):

lm = handLm.landmark

wrist = lm[0]

indexBase = lm[5]

scaleFactor = findDistance(wrist, indexBase)

tips = [lm[4],lm[8],lm[12],lm[16],lm[20]]

tipDistances =[]

for tip in tips:

dist = findDistance( wrist, tip) / scaleFactor

tipDistances.append(dist)

for startTip in range(len(tips)):

for endTip in range(startTip+1, len(tips)):

dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor

tipDistances.append(dist)

return tipDistances

def calculateError(tipDist, trainDist):

totalError = 0

for idx in range(len(tipDist)):

diff = tipDist[idx] -trainDist[idx]

totalError = totalError + abs(diff)

return totalError

def findGesture(tipLive, trainedData, maxAllowedError):

bestMatch = "Unknown"

lowestError = 999999999

for gestureName, trainedDist in trainedData.items():

error = calculateError(tipLive, trainedDist)

if error < lowestError:

lowestError = error

bestMatch = gestureName

if lowestError > maxAllowedError:

return "Unknown"

if lowestError <= maxAllowedError:

return bestMatch

print("HAND GESTURE TRAINING SETUP")

numGestures = int(input("Enter Number of Gestures to Train On: "))

gestureNames =[]

for idx in range(numGestures):

name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ")

gestureNames.append(name)

W=1280

H=720

tStart = time.time()

fps = 15

RES = (W,H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.002

fontColor = (0,0,255)

textFps = (int(W*.01),int(H*.07))

textGesture= (int(W*.01),int(H*.14))

textGesturePrompt = (int(W*.01),int(H*.21))

maxErrorThreshold = 3.5

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

trainedData ={}

trainIdx = 0

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

frameHeight, frameWidth, _ = frame.shape

for handLm in results.multi_hand_landmarks:

# Draw landmark numbers on hand joints

for idx, landmark in enumerate(handLm.landmark):

pixelX = int(landmark.x * frameWidth)

pixelY = int(landmark.y * frameHeight)

cv2.putText(

frame, str(idx), (pixelX, pixelY),

cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness

)

if trainIdx < len(gestureNames):

targetName = gestureNames[trainIdx]

promptText = "show " + targetName + " & Press Space"

cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness)

key = cv2.waitKey(1)

if key == ord(' '):

if results.multi_hand_landmarks:

targetHand = results.multi_hand_landmarks[0]

trainedData[targetName] = getTipDistances(targetHand)

trainIdx = trainIdx + 1

if key == ord('q'):

break

detectGesture = "Unknown"

if results.multi_hand_landmarks:

liveDistances = getTipDistances(results.multi_hand_landmarks[0])

detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

labelText = "Gesture: " + detectGesture

cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 49: Live Animated Avatars on the SSD1306 OLED With OpenCV

September 10, 2026 admin

Hey everybody! Welcome back to the AI on the Edge series!

In Lesson 49, we’re doing something that’s just plain cool — we’re turning that tiny little SSD1306 OLED into a live animated avatar that follows your face and hands in real time!

That’s right — no more static images, and no more being tethered to your desktop and monitor. This time your OLED is alive! Move your head, smile, raise your eyebrows, wave your hands… and the little display instantly mirrors your movements like a miniature digital twin. It’s like having a tiny version of yourself living on that 128×64 screen!

What You’ll Learn in This Lesson:

How to use MediaPipe Face Mesh to track 468 facial landmarks
How to combine it with Hand Tracking at the same time
The trick to projecting those landmarks onto a tiny OLED in real time
How to draw both points and connecting lines to create smooth, animated avatars
How to keep everything running fast on the Raspberry Pi

This is one of those projects that makes people stop and say “Wait… how is that even possible on that little screen?!”

By the end of this lesson, you’ll have a foundation for building all kinds of fun real-time avatar projects — everything from simple face mirrors to gesture-controlled characters and beyond.

Why This Is Awesome:

We’re not just displaying video on the big screen anymore — we’re compressing all that AI power down onto a cheap little OLED. This is real AI on the Edge stuff, and it looks incredibly impressive for how simple the hardware is.

Whether you want to build interactive displays, robot faces, wearable tech, or just blow your friends’ minds, this lesson gives you the core technique to make it happen.

Ready to make your OLED come alive?

Fire up the code, run it, and watch your tiny digital self start moving with you. Then start experimenting! Try drawing just the eyes and mouth, or just the hands, or maybe even turn it into a little stick figure avatar.

As always, I want to see what you create with this! Drop your versions in the comments or tag me — I love seeing the creative stuff you guys come up with. Don’t just copy and paste my code. Show you really understand the video by making the code yout own.

Enough talk, lets get this party started!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================

import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import numpy as np
import board
import adafruit_ssd1306
from PIL import Image, ImageDraw, ImageFont
import time

wOLED = 128
hOLED = 64
I2C_ADDRESS = 0x3C
i2c = board.I2C()
oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

W=1280
H=720
tStart = time.time()
fps = 0

piCam = Picamera2(1)
piCam.preview_configuration.main.size = (W, H)
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate = 60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

faceMesh = mp.solutions.face_mesh.FaceMesh(
    
    max_num_faces=1,
    refine_landmarks=True,
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5)

hands = mp.solutions.hands.Hands(
    model_complexity = 0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands=2 )

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)
while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame = piCam.capture_array()
    frame = cv2.flip(frame, -1)

    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    faceResults = faceMesh.process(rgb)
    handResults = hands.process(rgb)
    
    image = Image.new("1",(wOLED,hOLED))
    draw = ImageDraw.Draw(image)
    
    if handResults.multi_hand_landmarks:
        for myHands in handResults.multi_hand_landmarks:
            landMarks = myHands.landmark
            lmAll =[]
            for lm in landMarks:
                x = int(lm.x*W)
                y = int(lm.y*H)
                cv2.circle(frame,(x,y),10,(255,0,0),-1)
                xOLED = int(lm.x*wOLED)
                yOLED = int(lm.y*hOLED)
                draw.point((xOLED,yOLED), fill =255)
                lmAll.append((xOLED,yOLED))
            #Uncomment out these lines if you want
            #Lines between the finger joints
#             for idx in mp.solutions.hands.HAND_CONNECTIONS:
#                 lineStart = idx[0]
#                 lineEnd = idx[1]
#                 pt1 = lmAll[lineStart]
#                 pt2 = lmAll[lineEnd]
#                 draw.line((pt1[0],pt1[1],pt2[0],pt2[1]),width=1,fill=255)
#                              
    if faceResults.multi_face_landmarks:
        for faceLandMarks in faceResults.multi_face_landmarks:
            #print(faceLandmarks)
            lmAll = []
            for lm in faceLandMarks.landmark:
                xOLED = int(lm.x*wOLED)
                yOLED = int(lm.y*hOLED)
                draw.point((xOLED,yOLED), fill = 255)
                #print(xOLED,yOLED)
                lmAll.append((xOLED,yOLED))
            for idx in mp.solutions.face_mesh.FACEMESH_CONTOURS:
                #print(idx)
                lineStart = idx[0]
                lineEnd = idx[1]
                pt1 = lmAll[lineStart]
                pt2 = lmAll[lineEnd]
                draw.line((pt1[0],pt1[1],pt2[0],pt2[1]), width=1, fill = 255)



    oled.image(image)
    oled.show()


    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

    cv2.imshow("Camera", frame)

    if cv2.waitKey(1) == ord('q'):
        break

cv2.destroyAllWindows()
piCam.stop()
oled.fill(0)
oled.show()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import numpy as np

import board

import adafruit_ssd1306

from PIL import Image, ImageDraw, ImageFont

import time

wOLED = 128

hOLED = 64

I2C_ADDRESS = 0x3C

i2c = board.I2C()

oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

W=1280

H=720

tStart = time.time()

fps = 0

piCam = Picamera2(1)

piCam.preview_configuration.main.size = (W, H)

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate = 60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

faceMesh = mp.solutions.face_mesh.FaceMesh(

max_num_faces=1,

refine_landmarks=True,

min_detection_confidence=0.5,

min_tracking_confidence=0.5)

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands=2 )

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame = piCam.capture_array()

frame = cv2.flip(frame, -1)

rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

faceResults = faceMesh.process(rgb)

handResults = hands.process(rgb)

image = Image.new("1",(wOLED,hOLED))

draw = ImageDraw.Draw(image)

if handResults.multi_hand_landmarks:

for myHands in handResults.multi_hand_landmarks:

landMarks = myHands.landmark

lmAll =[]

for lm in landMarks:

x = int(lm.x*W)

y = int(lm.y*H)

cv2.circle(frame,(x,y),10,(255,0,0),-1)

xOLED = int(lm.x*wOLED)

yOLED = int(lm.y*hOLED)

draw.point((xOLED,yOLED), fill =255)

lmAll.append((xOLED,yOLED))

#Uncomment out these lines if you want

#Lines between the finger joints

# for idx in mp.solutions.hands.HAND_CONNECTIONS:

# lineStart = idx[0]

# lineEnd = idx[1]

# pt1 = lmAll[lineStart]

# pt2 = lmAll[lineEnd]

# draw.line((pt1[0],pt1[1],pt2[0],pt2[1]),width=1,fill=255)

if faceResults.multi_face_landmarks:

for faceLandMarks in faceResults.multi_face_landmarks:

#print(faceLandmarks)

lmAll = []

for lm in faceLandMarks.landmark:

xOLED = int(lm.x*wOLED)

yOLED = int(lm.y*hOLED)

draw.point((xOLED,yOLED), fill = 255)

#print(xOLED,yOLED)

lmAll.append((xOLED,yOLED))

for idx in mp.solutions.face_mesh.FACEMESH_CONTOURS:

#print(idx)

lineStart = idx[0]

lineEnd = idx[1]

pt1 = lmAll[lineStart]

pt2 = lmAll[lineEnd]

draw.line((pt1[0],pt1[1],pt2[0],pt2[1]), width=1, fill = 255)

oled.image(image)

oled.show()

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

if cv2.waitKey(1) == ord('q'):

break

cv2.destroyAllWindows()

piCam.stop()

oled.fill(0)

oled.show()

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 48: Hand Detection in OpenCV and MediaPipe on the Raspberry Pi

September 6, 2026 admin

In Lesson 48 we’re taking a huge step forward — we’re bringing real-time hand detection to the Raspberry Pi using the incredible power of MediaPipe combined with OpenCV and the Pi Camera. This is one of those projects that feels like pure magic when you see it working: the camera picks up your hands instantly, tracks all 21 landmarks on each hand, draws beautiful connections in real time, and does it all right on the edge with no cloud required.

You’ll learn how to set up MediaPipe’s Hands solution for reliable multi-hand tracking, how to efficiently process frames from the Picamera2 library, flip and convert images for proper display, and draw professional-looking landmarks and connections with custom styling. We also keep a smooth FPS counter running so you can see exactly how well your Pi is performing.

This lesson is exciting because hand tracking is the foundation for so many advanced gesture-control projects — think touchless interfaces, sign language recognition, robotic control, virtual instruments, and interactive art installations. Once you have solid hand detection running smoothly on your Raspberry Pi, the creative possibilities are almost endless.

So grab your Raspberry Pi, fire up the camera, and let’s get those hands dancing on the screen! As always, the full code is waiting for you below. Watch the video, follow along, and then start experimenting — I can’t wait to see what you build with this!

Let’s make something awesome!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================

import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)
mpDraw = mp.solutions.drawing_utils

handStyle = mpDraw.DrawingSpec(
    color = (100, 0, 255),
    thickness = 6,
    circle_radius = 8)
connectionStyle = mpDraw.DrawingSpec(
    color = (0,255,0),
    thickness = 5)

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    if results.multi_hand_landmarks:
        for handLM in results.multi_hand_landmarks:
            mpDraw.draw_landmarks(
                frame,
                handLM,
                mp.solutions.hands.HAND_CONNECTIONS,
                handStyle,
                connectionStyle)
        
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

mpDraw = mp.solutions.drawing_utils

handStyle = mpDraw.DrawingSpec(

color = (100, 0, 255),

thickness = 6,

circle_radius = 8)

connectionStyle = mpDraw.DrawingSpec(

color = (0,255,0),

thickness = 5)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

for handLM in results.multi_hand_landmarks:

mpDraw.draw_landmarks(

frame,

handLM,

mp.solutions.hands.HAND_CONNECTIONS,

handStyle,

connectionStyle)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

Technology Tutorials

Tag Archives: Fusion AI Lab Kit

AI on the Edge LESSON 55: Object Detection on the Raspberry Pi 5 With MediaPipe

What We Are Doing in Lesson 55

Key Takeaways & Concepts Covered

What You Will Need

Homework Assignment

AI on the Edge LESSON 53: Gesture Recognition on the Pi 5 with Training File

AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5

AI on the Edge LESSON 49: Live Animated Avatars on the SSD1306 OLED With OpenCV

What You’ll Learn in This Lesson:

Why This Is Awesome:

AI on the Edge LESSON 48: Hand Detection in OpenCV and MediaPipe on the Raspberry Pi

Making The World a Better Place One High Tech Project at a Time. Enjoy!