Tag Archives: AI on the Edge

AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5

September 28, 2026 admin

Welcome back, everyone! In today’s lesson, AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5, we are building a custom, real-time hand gesture recognition system from scratch. Up until now, we have been using MediaPipe to track keypoints, identify individual hand joints, and extract raw 3D coordinate data. But raw coordinates alone don’t tell the computer what your hand is actually doing. Today, we bridge that gap by teaching our Raspberry Pi 5 how to train on custom hand positions and dynamically classify live hand gestures using pure mathematics and vector geometry—no heavy neural network retraining required.

The core secret to making gesture recognition work reliably across different camera distances is scale-invariant distance normalization. If you simply measure the pixel distance between your fingertips and your wrist, bringing your hand closer to the camera lens will blow out those numbers and break your classifier. To solve this, our script establishes a baseline unit of measurement for every single frame: the distance between the wrist (landmark 0) and the index finger base joint (landmark 5). By dividing all measured finger-tip-to-wrist and tip-to-tip distances by this dynamic scale factor, the resulting feature vector remains identical whether your hand is two feet away or five feet away from the lens.

Once we extract this normalized 15-element feature vector—representing the spatial ratios between all five fingertips relative to the wrist and each other—we construct an interactive training phase directly inside the live video loop. The program prompts you in the terminal for the number of custom gestures you wish to record, along with their labels (such as “Peace”, “Thumbs Up”, or “Fist”). As you present each gesture to your camera feed and press the spacebar, the algorithm captures the instantaneous geometric signature of your hand and stores it in a runtime training dictionary.

During live inference, our classifier calculates the Sum of Absolute Errors (SAE), or Manhattan distance, between the live incoming feature vector and every stored profile in our training dataset. The function loops through all saved gestures to find the candidate with the absolute lowest cumulative error score. To prevent false positives when an untrained or messy hand pose is shown, we compare that lowest error score against a strict threshold value (maxErrorThreshold = 3.5). If the error falls within the allowable boundary, the matched gesture label is instantly displayed on the live OpenCV HUD; otherwise, the system safely defaults to “Unknown”.

We manage our camera feed using high-speed frame capture with Picamera2 running at 60 frames per second at 1280×720 resolution, processing hand detection with MediaPipe Hands, and rendering real-time performance diagnostics—including an exponential moving average FPS counter—directly onto the video output. Go ahead, study the methodology, load the concept onto your Raspberry Pi 5, grab a hot cup of coffee, and let’s get building!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import math

def findDistance(pt1, pt2):
    dX = pt1.x - pt2.x
    dY = pt1.y - pt2.y
    dZ = pt1.z - pt2.z
    dist = math.sqrt(dX*dX + dY*dY + dZ*dZ)
    return dist

def getTipDistances(handLm):
    lm = handLm.landmark
    wrist = lm[0]
    indexBase = lm[5]  
    scaleFactor = findDistance(wrist, indexBase)
    tips = [lm[4],lm[8],lm[12],lm[16],lm[20]]
    tipDistances =[]
    for tip in tips:
        dist = findDistance( wrist, tip) / scaleFactor
        tipDistances.append(dist)
    for startTip in range(len(tips)):
        for endTip in range(startTip+1, len(tips)):
            dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor
            tipDistances.append(dist)
    return tipDistances

def calculateError(tipDist, trainDist):
    totalError = 0
    for idx in range(len(tipDist)):
        diff = tipDist[idx] -trainDist[idx]
        totalError = totalError + abs(diff)
    return totalError

def findGesture(tipLive, trainedData, maxAllowedError):
    bestMatch = "Unknown"
    lowestError = 999999999
    for gestureName, trainedDist in trainedData.items():
        error  = calculateError(tipLive, trainedDist)
        if error < lowestError:
            lowestError = error
            bestMatch = gestureName
    if lowestError > maxAllowedError:
        return "Unknown"
    if lowestError <= maxAllowedError:
        return bestMatch
print("HAND GESTURE TRAINING SETUP")
numGestures = int(input("Enter Number of Gestures to Train On: "))
gestureNames =[]
for idx in range(numGestures):
    name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ")
    gestureNames.append(name)
W=1280
H=720
tStart = time.time()
fps = 15
RES = (W,H)
piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.002
fontColor = (0,0,255)

textFps = (int(W*.01),int(H*.07))
textGesture= (int(W*.01),int(H*.14))
textGesturePrompt = (int(W*.01),int(H*.21))

maxErrorThreshold = 3.5 

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)

trainedData ={}
trainIdx = 0

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    if results.multi_hand_landmarks:
        frameHeight, frameWidth, _ = frame.shape        
        for handLm in results.multi_hand_landmarks:
            # Draw landmark numbers on hand joints
            for idx, landmark in enumerate(handLm.landmark):
                pixelX = int(landmark.x * frameWidth)
                pixelY = int(landmark.y * frameHeight)
                cv2.putText(
                    frame, str(idx), (pixelX, pixelY),
                    cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness
                )
    if trainIdx < len(gestureNames):
        targetName = gestureNames[trainIdx]
        promptText = "show " + targetName + " & Press Space"
        cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness)
        key = cv2.waitKey(1)
        if key == ord(' '):
            if results.multi_hand_landmarks:
                targetHand = results.multi_hand_landmarks[0]
                trainedData[targetName] = getTipDistances(targetHand)
                trainIdx = trainIdx + 1
        if key == ord('q'):
            break
    detectGesture = "Unknown"
    if results.multi_hand_landmarks:
        liveDistances = getTipDistances(results.multi_hand_landmarks[0])
        detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)
    labelText = "Gesture: " + detectGesture
    cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness)
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import math

def findDistance(pt1, pt2):

dX = pt1.x - pt2.x

dY = pt1.y - pt2.y

dZ = pt1.z - pt2.z

dist = math.sqrt(dX*dX + dY*dY + dZ*dZ)

return dist

def getTipDistances(handLm):

lm = handLm.landmark

wrist = lm[0]

indexBase = lm[5]

scaleFactor = findDistance(wrist, indexBase)

tips = [lm[4],lm[8],lm[12],lm[16],lm[20]]

tipDistances =[]

for tip in tips:

dist = findDistance( wrist, tip) / scaleFactor

tipDistances.append(dist)

for startTip in range(len(tips)):

for endTip in range(startTip+1, len(tips)):

dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor

tipDistances.append(dist)

return tipDistances

def calculateError(tipDist, trainDist):

totalError = 0

for idx in range(len(tipDist)):

diff = tipDist[idx] -trainDist[idx]

totalError = totalError + abs(diff)

return totalError

def findGesture(tipLive, trainedData, maxAllowedError):

bestMatch = "Unknown"

lowestError = 999999999

for gestureName, trainedDist in trainedData.items():

error = calculateError(tipLive, trainedDist)

if error < lowestError:

lowestError = error

bestMatch = gestureName

if lowestError > maxAllowedError:

return "Unknown"

if lowestError <= maxAllowedError:

return bestMatch

print("HAND GESTURE TRAINING SETUP")

numGestures = int(input("Enter Number of Gestures to Train On: "))

gestureNames =[]

for idx in range(numGestures):

name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ")

gestureNames.append(name)

W=1280

H=720

tStart = time.time()

fps = 15

RES = (W,H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.002

fontColor = (0,0,255)

textFps = (int(W*.01),int(H*.07))

textGesture= (int(W*.01),int(H*.14))

textGesturePrompt = (int(W*.01),int(H*.21))

maxErrorThreshold = 3.5

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

trainedData ={}

trainIdx = 0

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

frameHeight, frameWidth, _ = frame.shape

for handLm in results.multi_hand_landmarks:

# Draw landmark numbers on hand joints

for idx, landmark in enumerate(handLm.landmark):

pixelX = int(landmark.x * frameWidth)

pixelY = int(landmark.y * frameHeight)

cv2.putText(

frame, str(idx), (pixelX, pixelY),

cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness

)

if trainIdx < len(gestureNames):

targetName = gestureNames[trainIdx]

promptText = "show " + targetName + " & Press Space"

cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness)

key = cv2.waitKey(1)

if key == ord(' '):

if results.multi_hand_landmarks:

targetHand = results.multi_hand_landmarks[0]

trainedData[targetName] = getTipDistances(targetHand)

trainIdx = trainIdx + 1

if key == ord('q'):

break

detectGesture = "Unknown"

if results.multi_hand_landmarks:

liveDistances = getTipDistances(results.multi_hand_landmarks[0])

detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

labelText = "Gesture: " + detectGesture

cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

Hand Recognition on the Raspberry Pi 5 With openCV and Mediapipe

September 25, 2026 admin

This is a base program which allows you to use mediapipe on the Raspberry Pi 5 under openCV to identify each landmark on a hand. This program is important as we can see the index of each landmark. This will serve as the basis for our next program which will allow us to begin to do gesture recogntion.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp

W=1280
H=720
tStart = time.time()
fps = 15
RES = (W,H)
piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.002
fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    if results.multi_hand_landmarks:
        frameHeight, frameWidth, _ = frame.shape        
        for handLm in results.multi_hand_landmarks:
            # Draw landmark numbers on hand joints
            for idx, landmark in enumerate(handLm.landmark):
                pixelX = int(landmark.x * frameWidth)
                pixelY = int(landmark.y * frameHeight)
                cv2.putText(
                    frame, str(idx), (pixelX, pixelY),
                    cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness
                )
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

W=1280

H=720

tStart = time.time()

fps = 15

RES = (W,H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.002

fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

frameHeight, frameWidth, _ = frame.shape

for handLm in results.multi_hand_landmarks:

# Draw landmark numbers on hand joints

for idx, landmark in enumerate(handLm.landmark):

pixelX = int(landmark.x * frameWidth)

pixelY = int(landmark.y * frameHeight)

cv2.putText(

frame, str(idx), (pixelX, pixelY),

cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness

)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 50: Control NeoPixel Ring With Hand Gestures and MediaPipe

September 13, 2026 admin

Hey everybody, welcome back to AI on the Edge! In Lesson 50 we’re taking things to the next level — we’re combining real-time hand tracking with a physical NeoPixel ring so you can change colors just by pointing at the screen! Using MediaPipe on the Raspberry Pi, we track your index finger tip and turn the big camera view into an interactive color picker. Hover over any of the seven big colored circles (Red, Green, Blue, Cyan, Magenta, Yellow, or Black) and — boom — the NeoPixel ring instantly lights up with that color!

This lesson is super fun because it finally brings together everything we’ve been building: smooth hand detection, coordinate mapping between camera and OLED, visual feedback, and real-world hardware control. You’ll see your finger tracked with a big blue dot, watch the selected color update live on both the screen and the OLED, and control actual RGB lights with nothing but a gesture.

By the end of this lesson you’ll have a working gesture-controlled color mixer that feels like real interactive magic. This is the kind of project that makes people say “Wow, you built that on a Raspberry Pi?!”

This is also a perfect foundation for even cooler future projects — gesture-controlled lighting, touchless interfaces, interactive games, or even a full gesture-controlled robot or light show.

So grab your Raspberry Pi, hook up that NeoPixel ring, fire up the camera, and let’s start waving our hands around like we’re casting spells! As always, the complete code is right below the video. Watch along, run the code, then start tweaking it — maybe add more colors, make the ring react differently to gestures, or combine it with the hand connection drawing from earlier lessons.I’m really excited to see what you create with this one!Now go make something awesome!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import board
import adafruit_ssd1306
from PIL import Image,ImageDraw,ImageFont

wOLED = 128
hOLED = 64
I2C_ADDRESS = 0x3C
i2c = board.I2C()
oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

W=1280
H=720
tStart = time.time()
fps = 0

RES = (W,H)
piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)
mpDraw = mp.solutions.drawing_utils

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)

import neopixel_spi as neopixel
spi = board.SPI()
strip = neopixel.NeoPixel_SPI(spi, 12,
        pixel_order=neopixel.GRB,auto_write=False)
strip.fill(0)
strip.show()

xStart=int(W/2)
yStart=int(H/10)
margin=100
radius=25
radiusOutline=30
radiusFlash=35
boxW = 4*radius
boxH =int(.75*boxW)
xSpace= int((W-2*margin)/2/3)
colorDot= [(xStart-3*xSpace,yStart),(xStart-2*xSpace,yStart),(xStart-xSpace,yStart),(xStart,yStart),(xStart+xSpace,yStart),(xStart+2*xSpace,yStart),(xStart+3*xSpace,yStart)]
oledPOS=[]
for dot in colorDot:
    oledPOS.append((int(dot[0]/W*wOLED),int(dot[1]/H*hOLED*.4)))
oledText = ['R','G','B','C','M','Y','B']
colorLabel = 'B'
colors =[(0,0,255),(0,255,0),(255,0,0),(255,255,0),(255,0,255),(0,255,255),(0,0,0)]
xBox = xStart-3*xSpace
yBox = int(yStart*1.5)
boxUpperLeft = (int(xBox-radius-boxW/4),yBox)
boxLowerRight = (int(xBox-radius-boxW/4+boxW),yBox+boxH)
colorActive =(0,0,0)
inCircle = False
                 
while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    frame=cv2.flip(frame,1)
    image = Image.new('1',(wOLED,hOLED))
    draw = ImageDraw.Draw(image)
    for i in range(7):
        if inCircle:
            cv2.circle(frame,colorDot[i],radiusFlash,(255,255,255),-1)
        cv2.circle(frame,colorDot[i],radiusOutline,(200,200,200),-1)
        cv2.circle(frame,colorDot[i],radius,colors[i],-1)
        draw.text(oledPOS[i],oledText[i],fill=255)
    cv2.rectangle(frame,boxUpperLeft,boxLowerRight,colorActive,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    handResults = hands.process(rgbFrame)
    if handResults.multi_hand_landmarks:
        for myHands in handResults.multi_hand_landmarks:
            landMarks = myHands.landmark
            x = int(landMarks[8].x*W)
            y = int(landMarks[8].y*H)
            xOLED = int(landMarks[8].x*wOLED)
            yOLED = int(landMarks[8].y*hOLED)
            cv2.circle(frame,(x,y),10,(255,0,0), -1)
            draw.ellipse((xOLED-3,yOLED-4,xOLED+3,yOLED+4),fill=255)
            cnt = 0
            for pos in colorDot:
                if abs(x-pos[0])<radius and abs(y-pos[1])<radius:
                    colorActive=colors[cnt]
                    colorLabel =oledText[cnt]
                    #draw.text((int(wOLED/2),int(hOLED/2)),oledText[cnt],fill=255)
                    strip.fill((colorActive[2],colorActive[1],colorActive[0]))
                    strip.show()
                               
                cnt=cnt+1
    draw.text((int(wOLED/2),int(hOLED/2)),colorLabel,fill=255)                    
    oled.image(image)
    oled.show()
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import board

import adafruit_ssd1306

from PIL import Image,ImageDraw,ImageFont

wOLED = 128

hOLED = 64

I2C_ADDRESS = 0x3C

i2c = board.I2C()

oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

mpDraw = mp.solutions.drawing_utils

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

import neopixel_spi as neopixel

spi = board.SPI()

strip = neopixel.NeoPixel_SPI(spi, 12,

pixel_order=neopixel.GRB,auto_write=False)

strip.fill(0)

strip.show()

xStart=int(W/2)

yStart=int(H/10)

margin=100

radius=25

radiusOutline=30

radiusFlash=35

boxW = 4*radius

boxH =int(.75*boxW)

xSpace= int((W-2*margin)/2/3)

colorDot= [(xStart-3*xSpace,yStart),(xStart-2*xSpace,yStart),(xStart-xSpace,yStart),(xStart,yStart),(xStart+xSpace,yStart),(xStart+2*xSpace,yStart),(xStart+3*xSpace,yStart)]

oledPOS=[]

for dot in colorDot:

oledPOS.append((int(dot[0]/W*wOLED),int(dot[1]/H*hOLED*.4)))

oledText = ['R','G','B','C','M','Y','B']

colorLabel = 'B'

colors =[(0,0,255),(0,255,0),(255,0,0),(255,255,0),(255,0,255),(0,255,255),(0,0,0)]

xBox = xStart-3*xSpace

yBox = int(yStart*1.5)

boxUpperLeft = (int(xBox-radius-boxW/4),yBox)

boxLowerRight = (int(xBox-radius-boxW/4+boxW),yBox+boxH)

colorActive =(0,0,0)

inCircle = False

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

frame=cv2.flip(frame,1)

image = Image.new('1',(wOLED,hOLED))

draw = ImageDraw.Draw(image)

for i in range(7):

if inCircle:

cv2.circle(frame,colorDot[i],radiusFlash,(255,255,255),-1)

cv2.circle(frame,colorDot[i],radiusOutline,(200,200,200),-1)

cv2.circle(frame,colorDot[i],radius,colors[i],-1)

draw.text(oledPOS[i],oledText[i],fill=255)

cv2.rectangle(frame,boxUpperLeft,boxLowerRight,colorActive,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

handResults = hands.process(rgbFrame)

if handResults.multi_hand_landmarks:

for myHands in handResults.multi_hand_landmarks:

landMarks = myHands.landmark

x = int(landMarks[8].x*W)

y = int(landMarks[8].y*H)

xOLED = int(landMarks[8].x*wOLED)

yOLED = int(landMarks[8].y*hOLED)

cv2.circle(frame,(x,y),10,(255,0,0), -1)

draw.ellipse((xOLED-3,yOLED-4,xOLED+3,yOLED+4),fill=255)

cnt = 0

for pos in colorDot:

if abs(x-pos[0])<radius and abs(y-pos[1])<radius:

colorActive=colors[cnt]

colorLabel =oledText[cnt]

#draw.text((int(wOLED/2),int(hOLED/2)),oledText[cnt],fill=255)

strip.fill((colorActive[2],colorActive[1],colorActive[0]))

strip.show()

cnt=cnt+1

draw.text((int(wOLED/2),int(hOLED/2)),colorLabel,fill=255)

oled.image(image)

oled.show()

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

This is the core circuit we use for the class:

Fusion Hat Circuit Diagram — This is the circuit we will use moving forward in the class

This is the schematic to add the OLED:

Then this is the schematic to add the NeoPixel

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 44: Displaying Live MediaPipe FaceMesh Avatar on SSD1306 OLED in OpenCV

August 23, 2026 admin

This is Getting Crazy Cool!

Hey guys, welcome back to the Fusion AI Lab series, AI on the Edge! In Lesson 44 we just took things up another notch. We’re now running MediaPipe Face Mesh on our Raspberry Pi 5, pulling all 468 facial landmarks in real time, and then drawing a clean, smooth face contour directly onto a tiny 128×64 OLED display!

That’s right — your face is now living on that little monochrome OLED in real time! Every eyebrow raise, smile, head tilt, and eye movement gets faithfully reproduced on the display. We’re pulling the camera feed with picamera2, processing it with MediaPipe’s powerful face mesh, scaling the landmarks down to OLED resolution, and then drawing the official FACEMESH_CONTOURS connections using PIL. The result is surprisingly smooth and fun to watch! This project really shows the power of combining modern AI vision tools with simple embedded hardware. Watching your own face rendered in real time on a tiny OLED is just plain awesome — it feels like sci-fi stuff from just a few years ago, and now we’re doing it on the Edge with our Fusion AI Lab kit!

If you’ve been following along, you’re starting to see how powerful these tools are becoming. We’re no longer just detecting faces — we’re understanding the structure of the face and visualizing it however we want. And we’re just getting started!

So fire up your Raspberry Pi, grab that OLED, and let’s keep pushing the limits. You’re doing some seriously cool stuff!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import numpy as np
#################################
import board
import adafruit_ssd1306
from PIL import Image, ImageDraw, ImageFont
import time

wOLED = 128
hOLED = 64
I2C_ADDRESS = 0x3C
i2c = board.I2C()
oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

#################################
W=1280
H=720
tStart = time.time()
fps = 0

piCam = Picamera2(1)
piCam.preview_configuration.main.size = (W, H)
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate = 60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

faceMesh = mp.solutions.face_mesh.FaceMesh(
    
    max_num_faces=1,
    refine_landmarks=True,
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5)

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)
while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame = piCam.capture_array()
    frame = cv2.flip(frame, -1)

    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = faceMesh.process(rgb)
    
    if results.multi_face_landmarks:
        for faceLandMarks in results.multi_face_landmarks:
            #print(faceLandmarks)
            image = Image.new("1",(wOLED,hOLED))
            draw = ImageDraw.Draw(image)
            lmAll = []
            for lm in faceLandMarks.landmark:
                xOLED = int(lm.x*wOLED)
                yOLED = int(lm.y*hOLED)
                draw.point((xOLED,yOLED), fill = 255)
                #print(xOLED,yOLED)
                lmAll.append((xOLED,yOLED))
            for idx in mp.solutions.face_mesh.FACEMESH_CONTOURS:
                #print(idx)
                lineStart = idx[0]
                lineEnd = idx[1]
                pt1 = lmAll[lineStart]
                pt2 = lmAll[lineEnd]
                draw.line((pt1[0],pt1[1],pt2[0],pt2[1]), width=1, fill = 255)


            #print('XXXXXXXXXXXXXXXXXXx')
            oled.image(image)
            oled.show()
        #print()

#     if results.multi_face_landmarks:
#         for faceLandmarks in results.multi_face_landmarks:
#             # Optional second pass for glow effect
#             mp.solutions.drawing_utils.draw_landmarks(
#                 image=frame,
#                 landmark_list=faceLandmarks,
#                 connections=mp.solutions.face_mesh.FACEMESH_TESSELATION,
#                 landmark_drawing_spec=None,
#                 connection_drawing_spec=mp.solutions.drawing_utils.DrawingSpec(
#                     color=(100, 100, 100),
#                     thickness=1
#                 )
#             )
#             # === Best looking translucent mesh ===
#             mp.solutions.drawing_utils.draw_landmarks(
#                 image=frame,
#                 landmark_list=faceLandmarks,
#                 connections=mp.solutions.face_mesh.FACEMESH_CONTOURS,
#                 landmark_drawing_spec=None,                    # Hide dots
#                 connection_drawing_spec=mp.solutions.drawing_utils.DrawingSpec(
#                     color=(0, 0, 180),      # Bright cyan/teal
#                     thickness=7
#                 )
#             )
# 
#             # Optional: Extra detail on irises (makes eyes look better)
#             mp.solutions.drawing_utils.draw_landmarks(
#                 image=frame,
#                 landmark_list=faceLandmarks,
#                 connections=mp.solutions.face_mesh.FACEMESH_IRISES,
#                 landmark_drawing_spec=None,
#                 connection_drawing_spec=mp.solutions.drawing_utils.DrawingSpec(
#                     color=(255, 0, 0),
#                     thickness=7
#                 )
#             )

    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

    cv2.imshow("Camera", frame)

    if cv2.waitKey(1) == ord('q'):
        break

cv2.destroyAllWindows()
piCam.stop()
oled.fill(0)
oled.show()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import numpy as np

#################################

import board

import adafruit_ssd1306

from PIL import Image, ImageDraw, ImageFont

import time

wOLED = 128

hOLED = 64

I2C_ADDRESS = 0x3C

i2c = board.I2C()

oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

#################################

W=1280

H=720

tStart = time.time()

fps = 0

piCam = Picamera2(1)

piCam.preview_configuration.main.size = (W, H)

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate = 60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

faceMesh = mp.solutions.face_mesh.FaceMesh(

max_num_faces=1,

refine_landmarks=True,

min_detection_confidence=0.5,

min_tracking_confidence=0.5)

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame = piCam.capture_array()

frame = cv2.flip(frame, -1)

rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = faceMesh.process(rgb)

if results.multi_face_landmarks:

for faceLandMarks in results.multi_face_landmarks:

#print(faceLandmarks)

image = Image.new("1",(wOLED,hOLED))

draw = ImageDraw.Draw(image)

lmAll = []

for lm in faceLandMarks.landmark:

xOLED = int(lm.x*wOLED)

yOLED = int(lm.y*hOLED)

draw.point((xOLED,yOLED), fill = 255)

#print(xOLED,yOLED)

lmAll.append((xOLED,yOLED))

for idx in mp.solutions.face_mesh.FACEMESH_CONTOURS:

#print(idx)

lineStart = idx[0]

lineEnd = idx[1]

pt1 = lmAll[lineStart]

pt2 = lmAll[lineEnd]

draw.line((pt1[0],pt1[1],pt2[0],pt2[1]), width=1, fill = 255)

#print('XXXXXXXXXXXXXXXXXXx')

oled.image(image)

oled.show()

#print()

# if results.multi_face_landmarks:

# for faceLandmarks in results.multi_face_landmarks:

# # Optional second pass for glow effect

# mp.solutions.drawing_utils.draw_landmarks(

# image=frame,

# landmark_list=faceLandmarks,

# connections=mp.solutions.face_mesh.FACEMESH_TESSELATION,

# landmark_drawing_spec=None,

# connection_drawing_spec=mp.solutions.drawing_utils.DrawingSpec(

# color=(100, 100, 100),

# thickness=1

# )

# # === Best looking translucent mesh ===

# mp.solutions.drawing_utils.draw_landmarks(

# image=frame,

# landmark_list=faceLandmarks,

# connections=mp.solutions.face_mesh.FACEMESH_CONTOURS,

# landmark_drawing_spec=None, # Hide dots

# connection_drawing_spec=mp.solutions.drawing_utils.DrawingSpec(

# color=(0, 0, 180), # Bright cyan/teal

# thickness=7

# )

# # Optional: Extra detail on irises (makes eyes look better)

# mp.solutions.drawing_utils.draw_landmarks(

# image=frame,

# landmark_list=faceLandmarks,

# connections=mp.solutions.face_mesh.FACEMESH_IRISES,

# landmark_drawing_spec=None,

# connection_drawing_spec=mp.solutions.drawing_utils.DrawingSpec(

# color=(255, 0, 0),

# thickness=7

# )

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

if cv2.waitKey(1) == ord('q'):

break

cv2.destroyAllWindows()

piCam.stop()

oled.fill(0)

oled.show()

In this lesson we are still using our standard Fusion AI Lab Kit cirtuit from the earlier lessons:

We have also connected the OLED to the above circuit according to this diagram:

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 39: Understanding MediaPipe Data Structures

August 5, 2026 admin

In this video lesson I show you how to understand the data structures returned by MediaPipe. I show you how to peel the data structure back, to get at the useful information.

When you run face detection with MediaPipe, the results object it returns is not a normal dictionary or list. It is a special custom object called SolutionOutputs. The easiest way to explore it is to start by checking the main attribute: results.detections. This is a Python list that contains one entry for every face detected in the current frame. If no faces are found, results.detections will be None or an empty list.

To extract useful information, you loop through results.detections. Each item in that list is a Detection object. From this object, you can access two main things: the confidence score using detection.score[0], and the location data using detection.location_data. Inside location_data, you will find relative_bounding_box (which gives you xmin, ymin, width, and height as values between 0 and 1) and relative_keypoints (a list of 6 facial points such as eyes, nose, and mouth).

The standard method is to first get the frame’s height and width, then multiply the normalized values (like xmin and width) by the actual pixel dimensions of the image to convert them into usable pixel coordinates. You can then use these coordinates with OpenCV functions such as cv2.rectangle() for the box or cv2.circle() for the keypoints.

By using simple print(type()), print(dir()), and print() statements on results, results.detections, and individual detection objects, you can quickly discover the full structure. This step-by-step approach — starting from results → detections → individual detection → location_data — lets you reliably reach all the useful information MediaPipe provides.

Below is the code we developed in the video.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================

import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

faceFinder = mp.solutions.face_detection.FaceDetection(
    model_selection = 0,
    min_detection_confidence = .5)
mpDraw = mp.solutions.drawing_utils

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    
    rgbFrame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
    results = faceFinder.process(rgbFrame)
    if results.detections:
        for detection in results.detections:
            #print(detection.location_data.relative_bounding_box)
            #print(detection)
            bbox =detection.location_data.relative_bounding_box
            x = int(bbox.xmin*W)
            y = int(bbox.ymin*H)
            w = int(bbox.width*W)
            h = int(bbox.height*H)
            cv2.rectangle(frame,
                         (x,y),
                         (x+w,y+h),
                         (0,255,255),
                         3)
            keypoints =detection.location_data.relative_keypoints
            #print(keypoints)
            for kp in keypoints:
                kpX = int(kp.x*W)
                kpY = int(kp.y*H)
                cv2.circle(frame,
                           (kpX,kpY),
                           15,
                           (255,0,0),
                           -1)
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

faceFinder = mp.solutions.face_detection.FaceDetection(

model_selection = 0,

min_detection_confidence = .5)

mpDraw = mp.solutions.drawing_utils

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

results = faceFinder.process(rgbFrame)

if results.detections:

for detection in results.detections:

#print(detection.location_data.relative_bounding_box)

#print(detection)

bbox =detection.location_data.relative_bounding_box

x = int(bbox.xmin*W)

y = int(bbox.ymin*H)

w = int(bbox.width*W)

h = int(bbox.height*H)

cv2.rectangle(frame,

(x,y),

(x+w,y+h),

(0,255,255),

keypoints =detection.location_data.relative_keypoints

#print(keypoints)

for kp in keypoints:

kpX = int(kp.x*W)

kpY = int(kp.y*H)

cv2.circle(frame,

(kpX,kpY),

15,

(255,0,0),

-1)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

Technology Tutorials

Tag Archives: AI on the Edge

AI on the Edge LESSON 52: Gesture Recognition on the Raspberry Pi 5

Hand Recognition on the Raspberry Pi 5 With openCV and Mediapipe

AI on the Edge LESSON 50: Control NeoPixel Ring With Hand Gestures and MediaPipe

AI on the Edge LESSON 44: Displaying Live MediaPipe FaceMesh Avatar on SSD1306 OLED in OpenCV

AI on the Edge LESSON 39: Understanding MediaPipe Data Structures

Making The World a Better Place One High Tech Project at a Time. Enjoy!