Tag Archives: Mediapipe

Welcome back, everyone! In this lesson, we take edge AI interactivity to the next level. Up until now, we have learned how to track hand landmarks, extract pixel coordinates, and calculate distances using MediaPipe. Today, we put all those building blocks together into a self-contained, real-time hand gesture recognition system running on the Raspberry Pi 5.

We are not relying on a black-box pre-trained gesture model. Instead, we write our own mathematical feature extractor, train our custom hand poses on the fly, save those gesture profiles directly to disk, and map each gesture to a non-blocking lighting animation on a NeoPixel LED ring driven over SPI.

System Architecture Overview

The system operates in three distinct phases running simultaneously inside our main loop:

Normalized Feature Extraction: MediaPipe extracts 3D hand landmark coordinates. To make gesture detection scale-invariant and distance-invariant, we normalize the distances between the fingertips and the wrist against a reference palm dimension.
Nearest-Neighbor Gesture Matching: Live distance vectors are compared against stored gesture profiles using a mean absolute error algorithm. If the error falls below our maximum threshold, the gesture is identified; otherwise, it defaults to “Unknown.”
Non-Blocking Hardware Animation Dispatch: The recognized gesture string triggers corresponding pattern functions on the 12-LED NeoPixel ring via neopixel_spi. Animations use non-blocking time delta checks to preserve high camera preview frame rates.

Gesture-to-Hardware Animation Mapping

Our script contains predefined animation routines mapped to recognized gesture strings. Here is how each pose controls the LED ring:

Recognized Gesture	LED Animation Pattern	Visual Effect
`fist`	Red Pulse	Sinusoidal breathing effect fading in and out
`palm`	Solid Beacon	Full brightness white ring
`peace`	Dual Spinner	Dual green and blue dots rotating over a dim red background
`thumbsup`	Solid Success	Constant green feedback
`point`	Radar Sweep	A green trailing dot chasing around a red ring
`rock`	Cyan Flame	Sinusoidal green-cyan intensity modulation
`ok`	Rainbow Wheel	Rotating HSV spectrum cycle
`Unknown` / Unmatched	Clear	LEDs turn completely off

Interactive Keyboard Controls

While the camera loop is active, you can interact with the system using your keyboard in the preview window:

Spacebar: Captures the current hand pose’s normalized distance vector, assigns it to the active target gesture name, and saves the updated dataset to gestureNEOPIXEL.pkl.
t: Prompts the terminal to add a new custom gesture name to your training queue dynamically.
d: Prompts the terminal to delete a specific gesture profile from the saved dataset.
q: Clears the NeoPixel ring, closes the camera pipeline, saves training data, and terminates the program cleanly.

Hardware Wiring Reference

Ensure your Raspberry Pi 5 SPI interface is enabled before running the code (sudo raspi-config -> Interface Options -> SPI).

NeoPixel Data Input (DIN): Connect to GPIO 10 (SPI0 MOSI / Pin 19).
NeoPixel Ground: Connect to any GND pin on the Pi expansion header.
NeoPixel Power: Connect to an external 5V power supply (ensure a shared ground with the Raspberry Pi).

Homework Challenge

Now that you have a working gesture recognition system linked to hardware animations, try adding your own custom gestures! Train a brand-new gesture pose (like a “three-finger count”), write a corresponding non-blocking animation function in updateNeoPixels(), and share your video results.

Here is the code we developed in the video lesson:

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import math
import pickle
import os
import colorsys

# ====================================================================
# NEOPIXEL CONTROL INITIALIZATION (Adafruit neopixel_spi via SPI)
# ====================================================================

import board
import neopixel_spi as neopixel

NUM_LEDS = 12  # Set to match your NeoPixel ring size (e.g., 12)
PO = neopixel.GRB

# Initialize NeoPixel ring using SPI
spi = board.SPI()
strip = neopixel.NeoPixel_SPI(
    spi, 
    NUM_LEDS, 
    pixel_order=PO, 
    auto_write=False
)

# Turn off all LEDs on startup
strip.fill(0)
strip.show()

NEOPIXEL_ENABLED = True
print("[INFO] NeoPixel SPI initialized successfully.")

# Animation state variables (non-blocking updates)
animation_step = 0
lastAnimUpdate = time.time()
# ====================================================================

DATA_FILE = "gestureNEOPIXEL.pkl"

# Function to save the trained gesture dictionary to a file using pickle
def saveTrainedData(filename, data):
    with open(filename, 'wb') as f:
        pickle.dump(data, f)
    print("[INFO] Saved gesture data to " + str(filename))

# Function to load previously saved gesture data from a file
def loadTrainedData(filename):
    if os.path.exists(filename):
        with open(filename, 'rb') as f:
            data = pickle.load(f)
            print("[INFO] Loaded " + str(len(data)) + " gesture(s) from " + str(filename))
        return data
    return {}

# --- SETUP & INITIAL TRAINING ---
trainedData = loadTrainedData(DATA_FILE)
gestureNames = []

if not trainedData:
    print("HAND GESTURE TRAINING SETUP")
    numGestures = int(input("Enter Number of Gestures to Train On: "))
    for idx in range(numGestures):
        name = input("Enter Name for Gesture #" + str(idx + 1) + ": ")
        gestureNames.append(name)

def findDistance(pt1, pt2):
    dX = pt1.x - pt2.x
    dY = pt1.y - pt2.y
    dZ = pt1.z - pt2.z
    return math.sqrt(dX*dX + dY*dY + dZ*dZ)

def getTipDistances(handLm):
    lm = handLm.landmark
    wrist = lm[0]
    indexBase = lm[5]  
    scaleFactor = findDistance(wrist, indexBase)
    tips = [lm[4], lm[8], lm[12], lm[16], lm[20]]
    tipDistances = []
    for tip in tips:
        dist = findDistance(wrist, tip) / scaleFactor
        tipDistances.append(dist)
    for startTip in range(len(tips)):
        for endTip in range(startTip + 1, len(tips)):
            dist = findDistance(tips[startTip], tips[endTip]) / scaleFactor
            tipDistances.append(dist)
    return tipDistances

def calculateError(tipDist, trainDist):
    totalError = 0
    for idx in range(len(tipDist)):
        diff = tipDist[idx] - trainDist[idx]
        totalError += abs(diff)
    return totalError

def findGesture(tipLive, trainedData, maxAllowedError):
    bestMatch = "Unknown"
    lowestError = 999999999
    for gestureName, trainedDist in trainedData.items():
        error = calculateError(tipLive, trainedDist)
        if error < lowestError:
            lowestError = error
            bestMatch = gestureName
    if lowestError > maxAllowedError:
        return "Unknown"
    return bestMatch

# ====================================================================
# NEOPIXEL CONTROL (COLOR HELPER & ANIMATION DISPATCHER)
# ====================================================================
def hsv2rgb(h):
    """ Converts HSV float (0.0-1.0) to RGB integer tuple (0-255) """
    r, g, b = colorsys.hsv_to_rgb(h, 1, 1)
    return (int(r * 255), int(g * 255), int(b * 255))

def updateNeoPixels(gesture):
    """ Non-blocking LED pattern update based on the current detected gesture """
    global animation_step, lastAnimUpdate
    if not NEOPIXEL_ENABLED:
        return

    now = time.time()
    gestureClean = gesture.lower()
    pattern_matched = False

    # 1. FIST / CHARGE (Red pulse)
    if gestureClean == "fist":
        pulse = int(127 + 127 * math.sin(now * 5.0))
        strip.fill((pulse, 0, 0))
        strip.show()
        pattern_matched = True

    # 2. OPEN PALM / BEACON (Solid Bright White)
    if gestureClean == "palm":
        strip.fill((255, 255, 255))
        strip.show()
        pattern_matched = True

    # 3. PEACE SIGN / DUAL SPINNER (Rotating Blue/Cyan dots)
    if gestureClean == "peace":
        if now - lastAnimUpdate > 0.08:
            animation_step = (animation_step + 1) % NUM_LEDS
            lastAnimUpdate = now
        strip.fill((40, 0, 0))  # Background red
        pos1 = animation_step
        pos2 = (animation_step + (NUM_LEDS // 2)) % NUM_LEDS
        strip[pos1] = (0, 255, 0)   # Green spinner dot
        strip[pos2] = (0, 0, 255)   # Blue spinner dot
        strip.show()
        pattern_matched = True

    # 4. THUMBS UP / SUCCESS (Solid Green)
    if gestureClean == "thumbsup":
        strip.fill((0, 255, 0))
        strip.show()
        pattern_matched = True

    # 5. POINTING / CHASER (Radar sweep)
    if gestureClean in "point":
        if now - lastAnimUpdate > 0.06:
            animation_step = (animation_step + 1) % NUM_LEDS
            lastAnimUpdate = now
        strip.fill((255, 0, 0))
        strip[animation_step] = (0, 255, 0)
        strip.show()
        pattern_matched = True

    # 6. ROCK / FIRE (Sinusoidal Green-Cyan pulse)
    if gestureClean == "rock":
        step = int((now * 100) % 360)
        brightness = (math.sin(step * math.pi / 180) + 1) / 2
        g = int(255 * brightness)
        b = int(50 * brightness)
        strip.fill((0, g, b))
        strip.show()
        pattern_matched = True

    # 7. OK SIGN / RAINBOW WHEEL (Rotating RGB spectrum)
    if gestureClean in "ok":
        if now - lastAnimUpdate > 0.03:
            animation_step = (animation_step + 1) % 100
            lastAnimUpdate = now
        hue = animation_step / 100.0
        strip.fill(hsv2rgb(hue))
        strip.show()
        pattern_matched = True

    # DEFAULT / UNKNOWN (LEDs OFF)
    if not pattern_matched:
        strip.fill((0, 0, 0))
        strip.show()
# ====================================================================

W, H = 1280, 720
tStart = time.time()
fps = 15
RES = (W, H)

piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate = 60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W * .01), int(H * .07))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W / 425)
fontScale = H * .002
fontColor = (0, 0, 255)

textFps = (int(W * .01), int(H * .07))
textGesture = (int(W * .01), int(H * .14))
textGesturePrompt = (int(W * .01), int(H * .21))

maxErrorThreshold = 3.5 

hands = mp.solutions.hands.Hands(
    model_complexity=0,
    min_detection_confidence=.5,
    min_tracking_confidence=.5,
    max_num_hands=2
)

trainIdx = 0

while True:
    deltaT = time.time() - tStart
    tStart = time.time()
    fps = fps * .95 + (1 / deltaT) * .05
    
    frame = piCam.capture_array()
    frame = cv2.flip(frame, -1)
    rgbFrame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    
    if results.multi_hand_landmarks:
        frameHeight, frameWidth, _ = frame.shape        
        for handLm in results.multi_hand_landmarks:
            for idx, landmark in enumerate(handLm.landmark):
                pixelX = int(landmark.x * frameWidth)
                pixelY = int(landmark.y * frameHeight)
                cv2.putText(
                    frame, str(idx), (pixelX, pixelY),
                    cv2.FONT_HERSHEY_SIMPLEX, .6 * fontScale, (255, 0, 0), fontThickness
                )
                
    if trainIdx < len(gestureNames):
        targetName = gestureNames[trainIdx]
        promptText = "show " + targetName + " & Press Space"
        cv2.putText(frame, promptText, textGesturePrompt, fontFace, fontScale, (0, 255, 255), fontThickness)

    detectGesture = "Unknown"
    if results.multi_hand_landmarks:
        liveDistances = getTipDistances(results.multi_hand_landmarks[0])
        detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

    # ====================================================================
    # EXECUTE HARDWARE PATTERN UPDATE
    # ====================================================================
    updateNeoPixels(detectGesture)
    # ====================================================================

    labelText = "Gesture: " + detectGesture
    cv2.putText(frame, labelText, textGesture, fontFace, fontScale, (0, 255, 0), fontThickness)
    myText = "FPS: " + str(round(fps, 1))
    cv2.putText(frame, myText, textLowerLeft, fontFace, fontScale, fontColor, fontThickness)
    
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera", 0, 60)
    
    key = cv2.waitKey(1)
    
    # Save landmark measurements when Space is pressed
    if key == ord(' '):
        if results.multi_hand_landmarks:
            if trainIdx < len(gestureNames):
                targetHand = results.multi_hand_landmarks[0]
                targetName = gestureNames[trainIdx]
                trainedData[targetName] = getTipDistances(targetHand)
                saveTrainedData(DATA_FILE, trainedData)
                trainIdx += 1

    # 't' key allows adding a NEW gesture dynamically during runtime
    if key == ord('t'):
        newGestureName = input("Enter name for new gesture to train: ").strip()
        if newGestureName:
            gestureNames.append(newGestureName)

    # 'd' key deletes a gesture
    if key == ord('d'):
        delName = input("Enter name of gesture to remove: ").strip()
        if delName in trainedData:
            del trainedData[delName]
            saveTrainedData(DATA_FILE, trainedData)
            print("[INFO] Removed " + delName + " from trained data.")
            if delName in gestureNames:
                gestureNames.remove(delName)
        if delName not in trainedData:
            print("[WARNING] Gesture: " + delName + " not found in trained data.")
        trainIdx=min(trainIdx,len(gestureNames))
    if key == ord('q'):
        break

# Clean up hardware and exit
# ====================================================================
# CLEAR LEDS ON SHUTDOWN
# ====================================================================
if NEOPIXEL_ENABLED:
    strip.fill((0, 0, 0))
    strip.show()
# ====================================================================

cv2.destroyAllWindows()
saveTrainedData(DATA_FILE, trainedData)
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import math

import pickle

import os

import colorsys

# ====================================================================

# NEOPIXEL CONTROL INITIALIZATION (Adafruit neopixel_spi via SPI)

# ====================================================================

import board

import neopixel_spi as neopixel

NUM_LEDS = 12 # Set to match your NeoPixel ring size (e.g., 12)

PO = neopixel.GRB

# Initialize NeoPixel ring using SPI

spi = board.SPI()

strip = neopixel.NeoPixel_SPI(

spi,

NUM_LEDS,

pixel_order=PO,

auto_write=False

)

# Turn off all LEDs on startup

strip.fill(0)

strip.show()

NEOPIXEL_ENABLED = True

print("[INFO] NeoPixel SPI initialized successfully.")

# Animation state variables (non-blocking updates)

animation_step = 0

lastAnimUpdate = time.time()

# ====================================================================

DATA_FILE = "gestureNEOPIXEL.pkl"

# Function to save the trained gesture dictionary to a file using pickle

def saveTrainedData(filename, data):

with open(filename, 'wb') as f:

pickle.dump(data, f)

print("[INFO] Saved gesture data to " + str(filename))

# Function to load previously saved gesture data from a file

def loadTrainedData(filename):

if os.path.exists(filename):

with open(filename, 'rb') as f:

data = pickle.load(f)

print("[INFO] Loaded " + str(len(data)) + " gesture(s) from " + str(filename))

return data

return {}

# --- SETUP & INITIAL TRAINING ---

trainedData = loadTrainedData(DATA_FILE)

gestureNames = []

if not trainedData:

print("HAND GESTURE TRAINING SETUP")

numGestures = int(input("Enter Number of Gestures to Train On: "))

for idx in range(numGestures):

name = input("Enter Name for Gesture #" + str(idx + 1) + ": ")

gestureNames.append(name)

def findDistance(pt1, pt2):

dX = pt1.x - pt2.x

dY = pt1.y - pt2.y

dZ = pt1.z - pt2.z

return math.sqrt(dX*dX + dY*dY + dZ*dZ)

def getTipDistances(handLm):

lm = handLm.landmark

wrist = lm[0]

indexBase = lm[5]

scaleFactor = findDistance(wrist, indexBase)

tips = [lm[4], lm[8], lm[12], lm[16], lm[20]]

tipDistances = []

for tip in tips:

dist = findDistance(wrist, tip) / scaleFactor

tipDistances.append(dist)

for startTip in range(len(tips)):

for endTip in range(startTip + 1, len(tips)):

dist = findDistance(tips[startTip], tips[endTip]) / scaleFactor

tipDistances.append(dist)

return tipDistances

def calculateError(tipDist, trainDist):

totalError = 0

for idx in range(len(tipDist)):

diff = tipDist[idx] - trainDist[idx]

totalError += abs(diff)

return totalError

def findGesture(tipLive, trainedData, maxAllowedError):

bestMatch = "Unknown"

lowestError = 999999999

for gestureName, trainedDist in trainedData.items():

error = calculateError(tipLive, trainedDist)

if error < lowestError:

lowestError = error

bestMatch = gestureName

if lowestError > maxAllowedError:

return "Unknown"

return bestMatch

# ====================================================================

# NEOPIXEL CONTROL (COLOR HELPER & ANIMATION DISPATCHER)

# ====================================================================

def hsv2rgb(h):

""" Converts HSV float (0.0-1.0) to RGB integer tuple (0-255) """

r, g, b = colorsys.hsv_to_rgb(h, 1, 1)

return (int(r * 255), int(g * 255), int(b * 255))

def updateNeoPixels(gesture):

""" Non-blocking LED pattern update based on the current detected gesture """

global animation_step, lastAnimUpdate

if not NEOPIXEL_ENABLED:

return

now = time.time()

gestureClean = gesture.lower()

pattern_matched = False

# 1. FIST / CHARGE (Red pulse)

if gestureClean == "fist":

pulse = int(127 + 127 * math.sin(now * 5.0))

strip.fill((pulse, 0, 0))

strip.show()

pattern_matched = True

# 2. OPEN PALM / BEACON (Solid Bright White)

if gestureClean == "palm":

strip.fill((255, 255, 255))

strip.show()

pattern_matched = True

# 3. PEACE SIGN / DUAL SPINNER (Rotating Blue/Cyan dots)

if gestureClean == "peace":

if now - lastAnimUpdate > 0.08:

animation_step = (animation_step + 1) % NUM_LEDS

lastAnimUpdate = now

strip.fill((40, 0, 0)) # Background red

pos1 = animation_step

pos2 = (animation_step + (NUM_LEDS // 2)) % NUM_LEDS

strip[pos1] = (0, 255, 0) # Green spinner dot

strip[pos2] = (0, 0, 255) # Blue spinner dot

strip.show()

pattern_matched = True

# 4. THUMBS UP / SUCCESS (Solid Green)

if gestureClean == "thumbsup":

strip.fill((0, 255, 0))

strip.show()

pattern_matched = True

# 5. POINTING / CHASER (Radar sweep)

if gestureClean in "point":

if now - lastAnimUpdate > 0.06:

animation_step = (animation_step + 1) % NUM_LEDS

lastAnimUpdate = now

strip.fill((255, 0, 0))

strip[animation_step] = (0, 255, 0)

strip.show()

pattern_matched = True

# 6. ROCK / FIRE (Sinusoidal Green-Cyan pulse)

if gestureClean == "rock":

step = int((now * 100) % 360)

brightness = (math.sin(step * math.pi / 180) + 1) / 2

g = int(255 * brightness)

b = int(50 * brightness)

strip.fill((0, g, b))

strip.show()

pattern_matched = True

# 7. OK SIGN / RAINBOW WHEEL (Rotating RGB spectrum)

if gestureClean in "ok":

if now - lastAnimUpdate > 0.03:

animation_step = (animation_step + 1) % 100

lastAnimUpdate = now

hue = animation_step / 100.0

strip.fill(hsv2rgb(hue))

strip.show()

pattern_matched = True

# DEFAULT / UNKNOWN (LEDs OFF)

if not pattern_matched:

strip.fill((0, 0, 0))

strip.show()

# ====================================================================

W, H = 1280, 720

tStart = time.time()

fps = 15

RES = (W, H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate = 60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W * .01), int(H * .07))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W / 425)

fontScale = H * .002

fontColor = (0, 0, 255)

textFps = (int(W * .01), int(H * .07))

textGesture = (int(W * .01), int(H * .14))

textGesturePrompt = (int(W * .01), int(H * .21))

maxErrorThreshold = 3.5

hands = mp.solutions.hands.Hands(

model_complexity=0,

min_detection_confidence=.5,

min_tracking_confidence=.5,

max_num_hands=2

)

trainIdx = 0

while True:

deltaT = time.time() - tStart

tStart = time.time()

fps = fps * .95 + (1 / deltaT) * .05

frame = piCam.capture_array()

frame = cv2.flip(frame, -1)

rgbFrame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

frameHeight, frameWidth, _ = frame.shape

for handLm in results.multi_hand_landmarks:

for idx, landmark in enumerate(handLm.landmark):

pixelX = int(landmark.x * frameWidth)

pixelY = int(landmark.y * frameHeight)

cv2.putText(

frame, str(idx), (pixelX, pixelY),

cv2.FONT_HERSHEY_SIMPLEX, .6 * fontScale, (255, 0, 0), fontThickness

)

if trainIdx < len(gestureNames):

targetName = gestureNames[trainIdx]

promptText = "show " + targetName + " & Press Space"

cv2.putText(frame, promptText, textGesturePrompt, fontFace, fontScale, (0, 255, 255), fontThickness)

detectGesture = "Unknown"

if results.multi_hand_landmarks:

liveDistances = getTipDistances(results.multi_hand_landmarks[0])

detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

# ====================================================================

# EXECUTE HARDWARE PATTERN UPDATE

# ====================================================================

updateNeoPixels(detectGesture)

# ====================================================================

labelText = "Gesture: " + detectGesture

cv2.putText(frame, labelText, textGesture, fontFace, fontScale, (0, 255, 0), fontThickness)

myText = "FPS: " + str(round(fps, 1))

cv2.putText(frame, myText, textLowerLeft, fontFace, fontScale, fontColor, fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera", 0, 60)

key = cv2.waitKey(1)

# Save landmark measurements when Space is pressed

if key == ord(' '):

if results.multi_hand_landmarks:

if trainIdx < len(gestureNames):

targetHand = results.multi_hand_landmarks[0]

targetName = gestureNames[trainIdx]

trainedData[targetName] = getTipDistances(targetHand)

saveTrainedData(DATA_FILE, trainedData)

trainIdx += 1

# 't' key allows adding a NEW gesture dynamically during runtime

if key == ord('t'):

newGestureName = input("Enter name for new gesture to train: ").strip()

if newGestureName:

gestureNames.append(newGestureName)

# 'd' key deletes a gesture

if key == ord('d'):

delName = input("Enter name of gesture to remove: ").strip()

if delName in trainedData:

del trainedData[delName]

saveTrainedData(DATA_FILE, trainedData)

print("[INFO] Removed " + delName + " from trained data.")

if delName in gestureNames:

gestureNames.remove(delName)

if delName not in trainedData:

print("[WARNING] Gesture: " + delName + " not found in trained data.")

trainIdx=min(trainIdx,len(gestureNames))

if key == ord('q'):

break

# Clean up hardware and exit

# ====================================================================

# CLEAR LEDS ON SHUTDOWN

# ====================================================================

if NEOPIXEL_ENABLED:

strip.fill((0, 0, 0))

strip.show()

# ====================================================================

cv2.destroyAllWindows()

saveTrainedData(DATA_FILE, trainedData)

print('Program Terminated')

Homework & Next Steps

Try setting up the circuit and running the code on your Raspberry Pi 5. Observe how the frame rate behaves when driving both the camera preview, the hand-tracking model, and the I2C OLED display simultaneously. For your homework, extend this script to include a dynamic color slider or gesture-based brightness control!

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 54: Custom Hand Gesture Recognition and Dynamic NeoPixel Control

October 8, 2026 admin

Welcome back, everyone! This is Paul McWhorter from TopTechBoy.com, and welcome to Lesson 54 in our AI on the Edge series.

In this lesson, we are taking our hand gesture recognition system to the next level by connecting it directly to real physical hardware! We’re going to take the normalized vector distance calculations and MediaPipe hand-tracking models we’ve been building over the past few lessons, and use them to drive a 12-LED NeoPixel Ring via SPI on the Raspberry Pi in real time.

If you’re following along in the series, pull up a chair, get yourself a nice hot cup of iced coffee, and let’s get right into it!

What We Are Building Today

In previous lessons, we learned how to extract 3D landmark coordinates from MediaPipe, scale them by the palm length to achieve scale-invariant gesture tracking, and store those relative Euclidean distances into a pickle database (gestureNEOPIXEL.pkl).

In Lesson 54, we map those live gesture detections directly to non-blocking LED animation patterns:

Fist: Pulses a dynamic red “charge” sine wave.
Open Palm: Drives the NeoPixel ring at full intensity white beacon.
Peace Sign (✌️): Runs a dual-spinner pattern with opposing green and blue tracer LEDs over a low background red glow.
Thumbs Up: Illumination of a solid, bright green “success” status.
Point: Sweeps a fast green radar chaser dot across a red background array.
Rock / Metal (🤘): Generates a smooth, sinusoidal cyan-to-green energy pulse.
OK Sign: Cycles through a full spectrum RGB rainbow wheel.
Unknown / Default: Gracefully turns off all LEDs.

Hardware Setup & Wiring (SPI Interface)

To ensure high frame rates without flickering, we drive our NeoPixels over the SPI bus using the neopixel_spi library.

Wiring Diagram Guide

NeoPixel VCC: Connect to 5V power supply (ensure adequate power if driving high brightness).
NeoPixel GND: Common ground with the Raspberry Pi GND pin.
NeoPixel DIN (Data In): Connect to GPIO 10 (SPI0 MOSI) on the Raspberry Pi pinout (Physical Pin 19).

Note: Ensure SPI is enabled in your Raspberry Pi system configuration (sudo raspi-config -> Interfaces -> SPI).

⌨️ Interactive Keyboard Controls

While running the main video processing loop, you can dynamically manage your trained gestures directly from your keyboard without interrupting the camera feed:

SPACE: Capture and train/overwrite the current gesture showing on screen.
T: Prompt the terminal to type in and add a brand-new gesture name on the fly.
D: Prompt the terminal to remove/delete an existing gesture from the active dataset.
Q: Cleanly turn off all NeoPixel LEDs, save the updated gesture data, and terminate the program.

Complete Python Source Code

Below is the complete, fully commented Python script for Lesson 54.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import math
import pickle
import os
import colorsys

# ====================================================================
# NEOPIXEL CONTROL INITIALIZATION (Adafruit neopixel_spi via SPI)
# ====================================================================

import board
import neopixel_spi as neopixel

NUM_LEDS = 12  # Set to match your NeoPixel ring size (e.g., 12)
PO = neopixel.GRB

# Initialize NeoPixel ring using SPI
spi = board.SPI()
strip = neopixel.NeoPixel_SPI(
    spi, 
    NUM_LEDS, 
    pixel_order=PO, 
    auto_write=False
)

# Turn off all LEDs on startup
strip.fill(0)
strip.show()

NEOPIXEL_ENABLED = True
print("[INFO] NeoPixel SPI initialized successfully.")

# Animation state variables (non-blocking updates)
animation_step = 0
lastAnimUpdate = time.time()
# ====================================================================

DATA_FILE = "gestureNEOPIXEL.pkl"

# Function to save the trained gesture dictionary to a file using pickle
def saveTrainedData(filename, data):
    with open(filename, 'wb') as f:
        pickle.dump(data, f)
    print("[INFO] Saved gesture data to " + str(filename))

# Function to load previously saved gesture data from a file
def loadTrainedData(filename):
    if os.path.exists(filename):
        with open(filename, 'rb') as f:
            data = pickle.load(f)
            print("[INFO] Loaded " + str(len(data)) + " gesture(s) from " + str(filename))
        return data
    return {}

# --- SETUP & INITIAL TRAINING ---
trainedData = loadTrainedData(DATA_FILE)
gestureNames = []

if not trainedData:
    print("HAND GESTURE TRAINING SETUP")
    numGestures = int(input("Enter Number of Gestures to Train On: "))
    for idx in range(numGestures):
        name = input("Enter Name for Gesture #" + str(idx + 1) + ": ")
        gestureNames.append(name)

def findDistance(pt1, pt2):
    dX = pt1.x - pt2.x
    dY = pt1.y - pt2.y
    dZ = pt1.z - pt2.z
    return math.sqrt(dX*dX + dY*dY + dZ*dZ)

def getTipDistances(handLm):
    lm = handLm.landmark
    wrist = lm[0]
    indexBase = lm[5]  
    scaleFactor = findDistance(wrist, indexBase)
    tips = [lm[4], lm[8], lm[12], lm[16], lm[20]]
    tipDistances = []
    for tip in tips:
        dist = findDistance(wrist, tip) / scaleFactor
        tipDistances.append(dist)
    for startTip in range(len(tips)):
        for endTip in range(startTip + 1, len(tips)):
            dist = findDistance(tips[startTip], tips[endTip]) / scaleFactor
            tipDistances.append(dist)
    return tipDistances

def calculateError(tipDist, trainDist):
    totalError = 0
    for idx in range(len(tipDist)):
        diff = tipDist[idx] - trainDist[idx]
        totalError += abs(diff)
    return totalError

def findGesture(tipLive, trainedData, maxAllowedError):
    bestMatch = "Unknown"
    lowestError = 999999999
    for gestureName, trainedDist in trainedData.items():
        error = calculateError(tipLive, trainedDist)
        if error < lowestError:
            lowestError = error
            bestMatch = gestureName
    if lowestError > maxAllowedError:
        return "Unknown"
    return bestMatch

# ====================================================================
# NEOPIXEL CONTROL (COLOR HELPER & ANIMATION DISPATCHER)
# ====================================================================
def hsv2rgb(h):
    """ Converts HSV float (0.0-1.0) to RGB integer tuple (0-255) """
    r, g, b = colorsys.hsv_to_rgb(h, 1, 1)
    return (int(r * 255), int(g * 255), int(b * 255))

def updateNeoPixels(gesture):
    """ Non-blocking LED pattern update based on the current detected gesture """
    global animation_step, lastAnimUpdate
    if not NEOPIXEL_ENABLED:
        return

    now = time.time()
    gestureClean = gesture.lower()
    pattern_matched = False

    # 1. FIST / CHARGE (Red pulse)
    if gestureClean == "fist":
        pulse = int(127 + 127 * math.sin(now * 5.0))
        strip.fill((pulse, 0, 0))
        strip.show()
        pattern_matched = True

    # 2. OPEN PALM / BEACON (Solid Bright White)
    if gestureClean == "palm":
        strip.fill((255, 255, 255))
        strip.show()
        pattern_matched = True

    # 3. PEACE SIGN / DUAL SPINNER (Rotating Blue/Cyan dots)
    if gestureClean == "peace":
        if now - lastAnimUpdate > 0.08:
            animation_step = (animation_step + 1) % NUM_LEDS
            lastAnimUpdate = now
        strip.fill((40, 0, 0))  # Background red
        pos1 = animation_step
        pos2 = (animation_step + (NUM_LEDS // 2)) % NUM_LEDS
        strip[pos1] = (0, 255, 0)   # Green spinner dot
        strip[pos2] = (0, 0, 255)   # Blue spinner dot
        strip.show()
        pattern_matched = True

    # 4. THUMBS UP / SUCCESS (Solid Green)
    if gestureClean == "thumbsup":
        strip.fill((0, 255, 0))
        strip.show()
        pattern_matched = True

    # 5. POINTING / CHASER (Radar sweep)
    if gestureClean in "point":
        if now - lastAnimUpdate > 0.06:
            animation_step = (animation_step + 1) % NUM_LEDS
            lastAnimUpdate = now
        strip.fill((255, 0, 0))
        strip[animation_step] = (0, 255, 0)
        strip.show()
        pattern_matched = True

    # 6. ROCK / FIRE (Sinusoidal Green-Cyan pulse)
    if gestureClean == "rock":
        step = int((now * 100) % 360)
        brightness = (math.sin(step * math.pi / 180) + 1) / 2
        g = int(255 * brightness)
        b = int(50 * brightness)
        strip.fill((0, g, b))
        strip.show()
        pattern_matched = True

    # 7. OK SIGN / RAINBOW WHEEL (Rotating RGB spectrum)
    if gestureClean in "ok":
        if now - lastAnimUpdate > 0.03:
            animation_step = (animation_step + 1) % 100
            lastAnimUpdate = now
        hue = animation_step / 100.0
        strip.fill(hsv2rgb(hue))
        strip.show()
        pattern_matched = True

    # DEFAULT / UNKNOWN (LEDs OFF)
    if not pattern_matched:
        strip.fill((0, 0, 0))
        strip.show()
# ====================================================================

W, H = 1280, 720
tStart = time.time()
fps = 15
RES = (W, H)

piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate = 60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W * .01), int(H * .07))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W / 425)
fontScale = H * .002
fontColor = (0, 0, 255)

textFps = (int(W * .01), int(H * .07))
textGesture = (int(W * .01), int(H * .14))
textGesturePrompt = (int(W * .01), int(H * .21))

maxErrorThreshold = 3.5 

hands = mp.solutions.hands.Hands(
    model_complexity=0,
    min_detection_confidence=.5,
    min_tracking_confidence=.5,
    max_num_hands=2
)

trainIdx = 0

while True:
    deltaT = time.time() - tStart
    tStart = time.time()
    fps = fps * .95 + (1 / deltaT) * .05
    
    frame = piCam.capture_array()
    frame = cv2.flip(frame, -1)
    rgbFrame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    
    if results.multi_hand_landmarks:
        frameHeight, frameWidth, _ = frame.shape        
        for handLm in results.multi_hand_landmarks:
            for idx, landmark in enumerate(handLm.landmark):
                pixelX = int(landmark.x * frameWidth)
                pixelY = int(landmark.y * frameHeight)
                cv2.putText(
                    frame, str(idx), (pixelX, pixelY),
                    cv2.FONT_HERSHEY_SIMPLEX, .6 * fontScale, (255, 0, 0), fontThickness
                )
                
    if trainIdx < len(gestureNames):
        targetName = gestureNames[trainIdx]
        promptText = "show " + targetName + " & Press Space"
        cv2.putText(frame, promptText, textGesturePrompt, fontFace, fontScale, (0, 255, 255), fontThickness)

    detectGesture = "Unknown"
    if results.multi_hand_landmarks:
        liveDistances = getTipDistances(results.multi_hand_landmarks[0])
        detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

    # ====================================================================
    # EXECUTE HARDWARE PATTERN UPDATE
    # ====================================================================
    updateNeoPixels(detectGesture)
    # ====================================================================

    labelText = "Gesture: " + detectGesture
    cv2.putText(frame, labelText, textGesture, fontFace, fontScale, (0, 255, 0), fontThickness)
    myText = "FPS: " + str(round(fps, 1))
    cv2.putText(frame, myText, textLowerLeft, fontFace, fontScale, fontColor, fontThickness)
    
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera", 0, 60)
    
    key = cv2.waitKey(1)
    
    # Save landmark measurements when Space is pressed
    if key == ord(' '):
        if results.multi_hand_landmarks:
            if trainIdx < len(gestureNames):
                targetHand = results.multi_hand_landmarks[0]
                targetName = gestureNames[trainIdx]
                trainedData[targetName] = getTipDistances(targetHand)
                saveTrainedData(DATA_FILE, trainedData)
                trainIdx += 1

    # 't' key allows adding a NEW gesture dynamically during runtime
    if key == ord('t'):
        newGestureName = input("Enter name for new gesture to train: ").strip()
        if newGestureName:
            gestureNames.append(newGestureName)

    # 'd' key deletes a gesture
    if key == ord('d'):
        delName = input("Enter name of gesture to remove: ").strip()
        if delName in trainedData:
            del trainedData[delName]
            saveTrainedData(DATA_FILE, trainedData)
            print("[INFO] Removed " + delName + " from trained data.")
            if delName in gestureNames:
                gestureNames.remove(delName)
        if delName not in trainedData:
            print("[WARNING] Gesture: " + delName + " not found in trained data.")
        trainIdx=min(trainIdx,len(gestureNames))
    if key == ord('q'):
        break

# Clean up hardware and exit
# ====================================================================
# CLEAR LEDS ON SHUTDOWN
# ====================================================================
if NEOPIXEL_ENABLED:
    strip.fill((0, 0, 0))
    strip.show()
# ====================================================================

cv2.destroyAllWindows()
saveTrainedData(DATA_FILE, trainedData)
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import math

import pickle

import os

import colorsys

# ====================================================================

# NEOPIXEL CONTROL INITIALIZATION (Adafruit neopixel_spi via SPI)

# ====================================================================

import board

import neopixel_spi as neopixel

NUM_LEDS = 12 # Set to match your NeoPixel ring size (e.g., 12)

PO = neopixel.GRB

# Initialize NeoPixel ring using SPI

spi = board.SPI()

strip = neopixel.NeoPixel_SPI(

spi,

NUM_LEDS,

pixel_order=PO,

auto_write=False

)

# Turn off all LEDs on startup

strip.fill(0)

strip.show()

NEOPIXEL_ENABLED = True

print("[INFO] NeoPixel SPI initialized successfully.")

# Animation state variables (non-blocking updates)

animation_step = 0

lastAnimUpdate = time.time()

# ====================================================================

DATA_FILE = "gestureNEOPIXEL.pkl"

# Function to save the trained gesture dictionary to a file using pickle

def saveTrainedData(filename, data):

with open(filename, 'wb') as f:

pickle.dump(data, f)

print("[INFO] Saved gesture data to " + str(filename))

# Function to load previously saved gesture data from a file

def loadTrainedData(filename):

if os.path.exists(filename):

with open(filename, 'rb') as f:

data = pickle.load(f)

print("[INFO] Loaded " + str(len(data)) + " gesture(s) from " + str(filename))

return data

return {}

# --- SETUP & INITIAL TRAINING ---

trainedData = loadTrainedData(DATA_FILE)

gestureNames = []

if not trainedData:

print("HAND GESTURE TRAINING SETUP")

numGestures = int(input("Enter Number of Gestures to Train On: "))

for idx in range(numGestures):

name = input("Enter Name for Gesture #" + str(idx + 1) + ": ")

gestureNames.append(name)

def findDistance(pt1, pt2):

dX = pt1.x - pt2.x

dY = pt1.y - pt2.y

dZ = pt1.z - pt2.z

return math.sqrt(dX*dX + dY*dY + dZ*dZ)

def getTipDistances(handLm):

lm = handLm.landmark

wrist = lm[0]

indexBase = lm[5]

scaleFactor = findDistance(wrist, indexBase)

tips = [lm[4], lm[8], lm[12], lm[16], lm[20]]

tipDistances = []

for tip in tips:

dist = findDistance(wrist, tip) / scaleFactor

tipDistances.append(dist)

for startTip in range(len(tips)):

for endTip in range(startTip + 1, len(tips)):

dist = findDistance(tips[startTip], tips[endTip]) / scaleFactor

tipDistances.append(dist)

return tipDistances

def calculateError(tipDist, trainDist):

totalError = 0

for idx in range(len(tipDist)):

diff = tipDist[idx] - trainDist[idx]

totalError += abs(diff)

return totalError

def findGesture(tipLive, trainedData, maxAllowedError):

bestMatch = "Unknown"

lowestError = 999999999

for gestureName, trainedDist in trainedData.items():

error = calculateError(tipLive, trainedDist)

if error < lowestError:

lowestError = error

bestMatch = gestureName

if lowestError > maxAllowedError:

return "Unknown"

return bestMatch

# ====================================================================

# NEOPIXEL CONTROL (COLOR HELPER & ANIMATION DISPATCHER)

# ====================================================================

def hsv2rgb(h):

""" Converts HSV float (0.0-1.0) to RGB integer tuple (0-255) """

r, g, b = colorsys.hsv_to_rgb(h, 1, 1)

return (int(r * 255), int(g * 255), int(b * 255))

def updateNeoPixels(gesture):

""" Non-blocking LED pattern update based on the current detected gesture """

global animation_step, lastAnimUpdate

if not NEOPIXEL_ENABLED:

return

now = time.time()

gestureClean = gesture.lower()

pattern_matched = False

# 1. FIST / CHARGE (Red pulse)

if gestureClean == "fist":

pulse = int(127 + 127 * math.sin(now * 5.0))

strip.fill((pulse, 0, 0))

strip.show()

pattern_matched = True

# 2. OPEN PALM / BEACON (Solid Bright White)

if gestureClean == "palm":

strip.fill((255, 255, 255))

strip.show()

pattern_matched = True

# 3. PEACE SIGN / DUAL SPINNER (Rotating Blue/Cyan dots)

if gestureClean == "peace":

if now - lastAnimUpdate > 0.08:

animation_step = (animation_step + 1) % NUM_LEDS

lastAnimUpdate = now

strip.fill((40, 0, 0)) # Background red

pos1 = animation_step

pos2 = (animation_step + (NUM_LEDS // 2)) % NUM_LEDS

strip[pos1] = (0, 255, 0) # Green spinner dot

strip[pos2] = (0, 0, 255) # Blue spinner dot

strip.show()

pattern_matched = True

# 4. THUMBS UP / SUCCESS (Solid Green)

if gestureClean == "thumbsup":

strip.fill((0, 255, 0))

strip.show()

pattern_matched = True

# 5. POINTING / CHASER (Radar sweep)

if gestureClean in "point":

if now - lastAnimUpdate > 0.06:

animation_step = (animation_step + 1) % NUM_LEDS

lastAnimUpdate = now

strip.fill((255, 0, 0))

strip[animation_step] = (0, 255, 0)

strip.show()

pattern_matched = True

# 6. ROCK / FIRE (Sinusoidal Green-Cyan pulse)

if gestureClean == "rock":

step = int((now * 100) % 360)

brightness = (math.sin(step * math.pi / 180) + 1) / 2

g = int(255 * brightness)

b = int(50 * brightness)

strip.fill((0, g, b))

strip.show()

pattern_matched = True

# 7. OK SIGN / RAINBOW WHEEL (Rotating RGB spectrum)

if gestureClean in "ok":

if now - lastAnimUpdate > 0.03:

animation_step = (animation_step + 1) % 100

lastAnimUpdate = now

hue = animation_step / 100.0

strip.fill(hsv2rgb(hue))

strip.show()

pattern_matched = True

# DEFAULT / UNKNOWN (LEDs OFF)

if not pattern_matched:

strip.fill((0, 0, 0))

strip.show()

# ====================================================================

W, H = 1280, 720

tStart = time.time()

fps = 15

RES = (W, H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate = 60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W * .01), int(H * .07))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W / 425)

fontScale = H * .002

fontColor = (0, 0, 255)

textFps = (int(W * .01), int(H * .07))

textGesture = (int(W * .01), int(H * .14))

textGesturePrompt = (int(W * .01), int(H * .21))

maxErrorThreshold = 3.5

hands = mp.solutions.hands.Hands(

model_complexity=0,

min_detection_confidence=.5,

min_tracking_confidence=.5,

max_num_hands=2

)

trainIdx = 0

while True:

deltaT = time.time() - tStart

tStart = time.time()

fps = fps * .95 + (1 / deltaT) * .05

frame = piCam.capture_array()

frame = cv2.flip(frame, -1)

rgbFrame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

frameHeight, frameWidth, _ = frame.shape

for handLm in results.multi_hand_landmarks:

for idx, landmark in enumerate(handLm.landmark):

pixelX = int(landmark.x * frameWidth)

pixelY = int(landmark.y * frameHeight)

cv2.putText(

frame, str(idx), (pixelX, pixelY),

cv2.FONT_HERSHEY_SIMPLEX, .6 * fontScale, (255, 0, 0), fontThickness

)

if trainIdx < len(gestureNames):

targetName = gestureNames[trainIdx]

promptText = "show " + targetName + " & Press Space"

cv2.putText(frame, promptText, textGesturePrompt, fontFace, fontScale, (0, 255, 255), fontThickness)

detectGesture = "Unknown"

if results.multi_hand_landmarks:

liveDistances = getTipDistances(results.multi_hand_landmarks[0])

detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

# ====================================================================

# EXECUTE HARDWARE PATTERN UPDATE

# ====================================================================

updateNeoPixels(detectGesture)

# ====================================================================

labelText = "Gesture: " + detectGesture

cv2.putText(frame, labelText, textGesture, fontFace, fontScale, (0, 255, 0), fontThickness)

myText = "FPS: " + str(round(fps, 1))

cv2.putText(frame, myText, textLowerLeft, fontFace, fontScale, fontColor, fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera", 0, 60)

key = cv2.waitKey(1)

# Save landmark measurements when Space is pressed

if key == ord(' '):

if results.multi_hand_landmarks:

if trainIdx < len(gestureNames):

targetHand = results.multi_hand_landmarks[0]

targetName = gestureNames[trainIdx]

trainedData[targetName] = getTipDistances(targetHand)

saveTrainedData(DATA_FILE, trainedData)

trainIdx += 1

# 't' key allows adding a NEW gesture dynamically during runtime

if key == ord('t'):

newGestureName = input("Enter name for new gesture to train: ").strip()

if newGestureName:

gestureNames.append(newGestureName)

# 'd' key deletes a gesture

if key == ord('d'):

delName = input("Enter name of gesture to remove: ").strip()

if delName in trainedData:

del trainedData[delName]

saveTrainedData(DATA_FILE, trainedData)

print("[INFO] Removed " + delName + " from trained data.")

if delName in gestureNames:

gestureNames.remove(delName)

if delName not in trainedData:

print("[WARNING] Gesture: " + delName + " not found in trained data.")

trainIdx=min(trainIdx,len(gestureNames))

if key == ord('q'):

break

# Clean up hardware and exit

# ====================================================================

# CLEAR LEDS ON SHUTDOWN

# ====================================================================

if NEOPIXEL_ENABLED:

strip.fill((0, 0, 0))

strip.show()

# ====================================================================

cv2.destroyAllWindows()

saveTrainedData(DATA_FILE, trainedData)

print('Program Terminated')

Homework Assignment

Your homework for this lesson is to modify the updateNeoPixels() function to add two new custom hand gestures of your own creation! Try implementing custom animations like a warm candle flicker or a pulsating dual-color pattern. Leave a link to your solution videos in the comments below!

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 53: Gesture Recognition on the Pi 5 with Training File

October 4, 2026 admin

Welcome back, everyone! In this lesson, we are taking our custom MediaPipe hand gesture recognition system to the next level. Up until now, every time you stopped your Python script, all of your hard-earned gesture training data vanished into thin air. If you wanted to run your project again, you had to re-train every single hand sign from scratch.

That ends today. We are going to make our AI remember what it learned by saving our gesture feature vectors to a local file using Python’s pickle library.

What You Will Learn in This Lesson

Data Persistence: How to serialize and save Python dictionaries containing landmark distance vectors directly to disk (gesture_data.pkl).
Automated Setup: How to check for existing training files at boot so your program automatically skips the initial training prompt if data already exists.
Dynamic Training Management: How to interactively train new gestures on the fly ('t' key) or delete outdated gestures ('d' key) during runtime while automatically updating the saved file.
Robust Error Handling: Ensuring index tracking (trainIdx) safely syncs with live gesture arrays so pressing the spacebar never crashes your video feed.

Key Concepts Explained

1. Pickling Data Objects In MediaPipe gesture recognition, a trained gesture isn’t a complex neural network weight file; it’s simply a dictionary mapping a gesture name (string) to a list of normalized keypoint ratios (floating-point numbers). Python’s pickle module allows us to convert this dictionary into a byte stream and write it directly to a file (.pkl). When the program launches, pickle.load() reconstructs the dictionary in milliseconds.

2. Managing Execution Flow on Startup When gesture_data.pkl exists, our program automatically populates gestureNames with the keys from trainedData and sets trainIdx past the end of the array. This keeps the camera pipeline running in Inference Mode immediately. If no file exists, the script seamlessly drops into Training Setup Mode to prompt the user for initial inputs.

This is the code we develop in this video lesson:

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import math
import os
import pickle

DATA_FILE = "gesture_data.pkl"

def saveTrainedData(filename, data):
    with open(filename, 'wb') as f:
        pickle.dump(data,f)
    print("[INFO] Saved Gesture Data to "+str(filename))
def loadTrainedData(filename):
    if os.path.exists(filename):
        with open(filename, 'rb') as f:
            data = pickle.load(f)
            print("[INFO] LOADED "+str(len(data))+ " gestures from " + str(filename))
            return data
    if not os.path.exists(filename):
        return {}

def findDistance(pt1, pt2):
    dX = pt1.x - pt2.x
    dY = pt1.y - pt2.y
    dZ = pt1.z - pt2.z
    dist = math.sqrt(dX*dX + dY*dY + dZ*dZ)
    return dist

def getTipDistances(handLm):
    lm = handLm.landmark
    wrist = lm[0]
    indexBase = lm[5]  
    scaleFactor = findDistance(wrist, indexBase)
    tips = [lm[4],lm[8],lm[12],lm[16],lm[20]]
    tipDistances =[]
    for tip in tips:
        dist = findDistance( wrist, tip) / scaleFactor
        tipDistances.append(dist)
    for startTip in range(len(tips)):
        for endTip in range(startTip+1, len(tips)):
            dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor
            tipDistances.append(dist)
    return tipDistances

def calculateError(tipDist, trainDist):
    totalError = 0
    for idx in range(len(tipDist)):
        diff = tipDist[idx] -trainDist[idx]
        totalError = totalError + abs(diff)
    return totalError

def findGesture(tipLive, trainedData, maxAllowedError):
    bestMatch = "Unknown"
    lowestError = 999999999
    for gestureName, trainedDist in trainedData.items():
        error  = calculateError(tipLive, trainedDist)
        if error < lowestError:
            lowestError = error
            bestMatch = gestureName
    if lowestError > maxAllowedError:
        return "Unknown"
    if lowestError <= maxAllowedError:
        return bestMatch
trainedData = loadTrainedData(DATA_FILE)
gestureNames =[]
if not trainedData:
    print("HAND GESTURE TRAINING SETUP")
    numGestures = int(input("Enter Number of Gestures to Train On: "))
    gestureNames =[]
    for idx in range(numGestures):
        name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ")
        gestureNames.append(name)
W=1280
H=720
tStart = time.time()
fps = 15
RES = (W,H)
piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.002
fontColor = (0,0,255)

textFps = (int(W*.01),int(H*.07))
textGesture= (int(W*.01),int(H*.14))
textGesturePrompt = (int(W*.01),int(H*.21))

maxErrorThreshold = 3.5 

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)
trainIdx = 0

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    if results.multi_hand_landmarks:
        frameHeight, frameWidth, _ = frame.shape        
        for handLm in results.multi_hand_landmarks:
            # Draw landmark numbers on hand joints
            for idx, landmark in enumerate(handLm.landmark):
                pixelX = int(landmark.x * frameWidth)
                pixelY = int(landmark.y * frameHeight)
                cv2.putText(
                    frame, str(idx), (pixelX, pixelY),
                    cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness
                )
    if trainIdx < len(gestureNames):
        targetName = gestureNames[trainIdx]
        promptText = "show " + targetName + " & Press Space"
        cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness)
    detectGesture = "Unknown"
    if results.multi_hand_landmarks:
        liveDistances = getTipDistances(results.multi_hand_landmarks[0])
        detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)
    labelText = "Gesture: " + detectGesture
    cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness)
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera",0,60)
    key = cv2.waitKey(1)
    if key == ord('t'):
        newGestureName = input("Enter Name for New Gesture: ")
        gestureNames.append(newGestureName)
        saveTrainedData(DATA_FILE, trainedData)
    if key == ord('d'):
        delName = input("Enter Name of Gesture to Remove: ")
        if delName not in trainedData:
            print("[WARNING] Gesture "+delName+ " Not Found in Training Data.")
        if delName in trainedData:
            del trainedData[delName]
            saveTrainedData(DATA_FILE,trainedData)
            print("[INFO] Removed "+delName+"from trained data.")
            if delName in gestureNames:
                gestureNames.remove(delName)
            trainIdx=min(trainIdx,len(gestureNames))
    if key == ord(' '):
        if results.multi_hand_landmarks:
            targetHand = results.multi_hand_landmarks[0]
            trainedData[targetName] = getTipDistances(targetHand)
            trainIdx = trainIdx + 1
    if key == ord('q'):
        break
cv2.destroyAllWindows()
saveTrainedData(DATA_FILE, trainedData)
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import math

import os

import pickle

DATA_FILE = "gesture_data.pkl"

def saveTrainedData(filename, data):

with open(filename, 'wb') as f:

pickle.dump(data,f)

print("[INFO] Saved Gesture Data to "+str(filename))

def loadTrainedData(filename):

if os.path.exists(filename):

with open(filename, 'rb') as f:

data = pickle.load(f)

print("[INFO] LOADED "+str(len(data))+ " gestures from " + str(filename))

return data

if not os.path.exists(filename):

return {}

def findDistance(pt1, pt2):

dX = pt1.x - pt2.x

dY = pt1.y - pt2.y

dZ = pt1.z - pt2.z

dist = math.sqrt(dX*dX + dY*dY + dZ*dZ)

return dist

def getTipDistances(handLm):

lm = handLm.landmark

wrist = lm[0]

indexBase = lm[5]

scaleFactor = findDistance(wrist, indexBase)

tips = [lm[4],lm[8],lm[12],lm[16],lm[20]]

tipDistances =[]

for tip in tips:

dist = findDistance( wrist, tip) / scaleFactor

tipDistances.append(dist)

for startTip in range(len(tips)):

for endTip in range(startTip+1, len(tips)):

dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor

tipDistances.append(dist)

return tipDistances

def calculateError(tipDist, trainDist):

totalError = 0

for idx in range(len(tipDist)):

diff = tipDist[idx] -trainDist[idx]

totalError = totalError + abs(diff)

return totalError

def findGesture(tipLive, trainedData, maxAllowedError):

bestMatch = "Unknown"

lowestError = 999999999

for gestureName, trainedDist in trainedData.items():

error = calculateError(tipLive, trainedDist)

if error < lowestError:

lowestError = error

bestMatch = gestureName

if lowestError > maxAllowedError:

return "Unknown"

if lowestError <= maxAllowedError:

return bestMatch

trainedData = loadTrainedData(DATA_FILE)

gestureNames =[]

if not trainedData:

print("HAND GESTURE TRAINING SETUP")

numGestures = int(input("Enter Number of Gestures to Train On: "))

gestureNames =[]

for idx in range(numGestures):

name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ")

gestureNames.append(name)

W=1280

H=720

tStart = time.time()

fps = 15

RES = (W,H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.07))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.002

fontColor = (0,0,255)

textFps = (int(W*.01),int(H*.07))

textGesture= (int(W*.01),int(H*.14))

textGesturePrompt = (int(W*.01),int(H*.21))

maxErrorThreshold = 3.5

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

trainIdx = 0

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

frameHeight, frameWidth, _ = frame.shape

for handLm in results.multi_hand_landmarks:

# Draw landmark numbers on hand joints

for idx, landmark in enumerate(handLm.landmark):

pixelX = int(landmark.x * frameWidth)

pixelY = int(landmark.y * frameHeight)

cv2.putText(

frame, str(idx), (pixelX, pixelY),

cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness

)

if trainIdx < len(gestureNames):

targetName = gestureNames[trainIdx]

promptText = "show " + targetName + " & Press Space"

cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness)

detectGesture = "Unknown"

if results.multi_hand_landmarks:

liveDistances = getTipDistances(results.multi_hand_landmarks[0])

detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold)

labelText = "Gesture: " + detectGesture

cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera",0,60)

key = cv2.waitKey(1)

if key == ord('t'):

newGestureName = input("Enter Name for New Gesture: ")

gestureNames.append(newGestureName)

saveTrainedData(DATA_FILE, trainedData)

if key == ord('d'):

delName = input("Enter Name of Gesture to Remove: ")

if delName not in trainedData:

print("[WARNING] Gesture "+delName+ " Not Found in Training Data.")

if delName in trainedData:

del trainedData[delName]

saveTrainedData(DATA_FILE,trainedData)

print("[INFO] Removed "+delName+"from trained data.")

if delName in gestureNames:

gestureNames.remove(delName)

trainIdx=min(trainIdx,len(gestureNames))

if key == ord(' '):

if results.multi_hand_landmarks:

targetHand = results.multi_hand_landmarks[0]

trainedData[targetName] = getTipDistances(targetHand)

trainIdx = trainIdx + 1

if key == ord('q'):

break

cv2.destroyAllWindows()

saveTrainedData(DATA_FILE, trainedData)

print('Program Terminated')

Homework Assignment

Now that your Pi 5 can remember hand gestures across reboots, it’s time to bridge software recognition with real-world hardware control!

The Challenge: Integrate an SPI NeoPixel Ring (WS2812B/SK6812) with your gesture recognition code.

Map distinct hand signs to dynamic LED patterns (for example: Fist = Red Pulse, Open Palm = Bright White Beacon, Peace Sign = Dual Spinner, OK Sign = Rainbow Wheel).
Make sure your animation updates are smooth and non-blocking so your OpenCV camera frame rate doesn’t drop!

We want to see who in the community can build the most impressive physical light demonstration controlled entirely by hand gestures. Share your video results and code modifications in the comments below!

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 50: Control NeoPixel Ring With Hand Gestures and MediaPipe

September 13, 2026 admin

Hey everybody, welcome back to AI on the Edge! In Lesson 50 we’re taking things to the next level — we’re combining real-time hand tracking with a physical NeoPixel ring so you can change colors just by pointing at the screen! Using MediaPipe on the Raspberry Pi, we track your index finger tip and turn the big camera view into an interactive color picker. Hover over any of the seven big colored circles (Red, Green, Blue, Cyan, Magenta, Yellow, or Black) and — boom — the NeoPixel ring instantly lights up with that color!

This lesson is super fun because it finally brings together everything we’ve been building: smooth hand detection, coordinate mapping between camera and OLED, visual feedback, and real-world hardware control. You’ll see your finger tracked with a big blue dot, watch the selected color update live on both the screen and the OLED, and control actual RGB lights with nothing but a gesture.

By the end of this lesson you’ll have a working gesture-controlled color mixer that feels like real interactive magic. This is the kind of project that makes people say “Wow, you built that on a Raspberry Pi?!”

This is also a perfect foundation for even cooler future projects — gesture-controlled lighting, touchless interfaces, interactive games, or even a full gesture-controlled robot or light show.

So grab your Raspberry Pi, hook up that NeoPixel ring, fire up the camera, and let’s start waving our hands around like we’re casting spells! As always, the complete code is right below the video. Watch along, run the code, then start tweaking it — maybe add more colors, make the ring react differently to gestures, or combine it with the hand connection drawing from earlier lessons.I’m really excited to see what you create with this one!Now go make something awesome!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import board
import adafruit_ssd1306
from PIL import Image,ImageDraw,ImageFont

wOLED = 128
hOLED = 64
I2C_ADDRESS = 0x3C
i2c = board.I2C()
oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

W=1280
H=720
tStart = time.time()
fps = 0

RES = (W,H)
piCam = Picamera2(1)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)
mpDraw = mp.solutions.drawing_utils

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)

import neopixel_spi as neopixel
spi = board.SPI()
strip = neopixel.NeoPixel_SPI(spi, 12,
        pixel_order=neopixel.GRB,auto_write=False)
strip.fill(0)
strip.show()

xStart=int(W/2)
yStart=int(H/10)
margin=100
radius=25
radiusOutline=30
radiusFlash=35
boxW = 4*radius
boxH =int(.75*boxW)
xSpace= int((W-2*margin)/2/3)
colorDot= [(xStart-3*xSpace,yStart),(xStart-2*xSpace,yStart),(xStart-xSpace,yStart),(xStart,yStart),(xStart+xSpace,yStart),(xStart+2*xSpace,yStart),(xStart+3*xSpace,yStart)]
oledPOS=[]
for dot in colorDot:
    oledPOS.append((int(dot[0]/W*wOLED),int(dot[1]/H*hOLED*.4)))
oledText = ['R','G','B','C','M','Y','B']
colorLabel = 'B'
colors =[(0,0,255),(0,255,0),(255,0,0),(255,255,0),(255,0,255),(0,255,255),(0,0,0)]
xBox = xStart-3*xSpace
yBox = int(yStart*1.5)
boxUpperLeft = (int(xBox-radius-boxW/4),yBox)
boxLowerRight = (int(xBox-radius-boxW/4+boxW),yBox+boxH)
colorActive =(0,0,0)
inCircle = False
                 
while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    frame=cv2.flip(frame,1)
    image = Image.new('1',(wOLED,hOLED))
    draw = ImageDraw.Draw(image)
    for i in range(7):
        if inCircle:
            cv2.circle(frame,colorDot[i],radiusFlash,(255,255,255),-1)
        cv2.circle(frame,colorDot[i],radiusOutline,(200,200,200),-1)
        cv2.circle(frame,colorDot[i],radius,colors[i],-1)
        draw.text(oledPOS[i],oledText[i],fill=255)
    cv2.rectangle(frame,boxUpperLeft,boxLowerRight,colorActive,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    handResults = hands.process(rgbFrame)
    if handResults.multi_hand_landmarks:
        for myHands in handResults.multi_hand_landmarks:
            landMarks = myHands.landmark
            x = int(landMarks[8].x*W)
            y = int(landMarks[8].y*H)
            xOLED = int(landMarks[8].x*wOLED)
            yOLED = int(landMarks[8].y*hOLED)
            cv2.circle(frame,(x,y),10,(255,0,0), -1)
            draw.ellipse((xOLED-3,yOLED-4,xOLED+3,yOLED+4),fill=255)
            cnt = 0
            for pos in colorDot:
                if abs(x-pos[0])<radius and abs(y-pos[1])<radius:
                    colorActive=colors[cnt]
                    colorLabel =oledText[cnt]
                    #draw.text((int(wOLED/2),int(hOLED/2)),oledText[cnt],fill=255)
                    strip.fill((colorActive[2],colorActive[1],colorActive[0]))
                    strip.show()
                               
                cnt=cnt+1
    draw.text((int(wOLED/2),int(hOLED/2)),colorLabel,fill=255)                    
    oled.image(image)
    oled.show()
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import board

import adafruit_ssd1306

from PIL import Image,ImageDraw,ImageFont

wOLED = 128

hOLED = 64

I2C_ADDRESS = 0x3C

i2c = board.I2C()

oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam = Picamera2(1)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

mpDraw = mp.solutions.drawing_utils

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

import neopixel_spi as neopixel

spi = board.SPI()

strip = neopixel.NeoPixel_SPI(spi, 12,

pixel_order=neopixel.GRB,auto_write=False)

strip.fill(0)

strip.show()

xStart=int(W/2)

yStart=int(H/10)

margin=100

radius=25

radiusOutline=30

radiusFlash=35

boxW = 4*radius

boxH =int(.75*boxW)

xSpace= int((W-2*margin)/2/3)

colorDot= [(xStart-3*xSpace,yStart),(xStart-2*xSpace,yStart),(xStart-xSpace,yStart),(xStart,yStart),(xStart+xSpace,yStart),(xStart+2*xSpace,yStart),(xStart+3*xSpace,yStart)]

oledPOS=[]

for dot in colorDot:

oledPOS.append((int(dot[0]/W*wOLED),int(dot[1]/H*hOLED*.4)))

oledText = ['R','G','B','C','M','Y','B']

colorLabel = 'B'

colors =[(0,0,255),(0,255,0),(255,0,0),(255,255,0),(255,0,255),(0,255,255),(0,0,0)]

xBox = xStart-3*xSpace

yBox = int(yStart*1.5)

boxUpperLeft = (int(xBox-radius-boxW/4),yBox)

boxLowerRight = (int(xBox-radius-boxW/4+boxW),yBox+boxH)

colorActive =(0,0,0)

inCircle = False

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

frame=cv2.flip(frame,1)

image = Image.new('1',(wOLED,hOLED))

draw = ImageDraw.Draw(image)

for i in range(7):

if inCircle:

cv2.circle(frame,colorDot[i],radiusFlash,(255,255,255),-1)

cv2.circle(frame,colorDot[i],radiusOutline,(200,200,200),-1)

cv2.circle(frame,colorDot[i],radius,colors[i],-1)

draw.text(oledPOS[i],oledText[i],fill=255)

cv2.rectangle(frame,boxUpperLeft,boxLowerRight,colorActive,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

handResults = hands.process(rgbFrame)

if handResults.multi_hand_landmarks:

for myHands in handResults.multi_hand_landmarks:

landMarks = myHands.landmark

x = int(landMarks[8].x*W)

y = int(landMarks[8].y*H)

xOLED = int(landMarks[8].x*wOLED)

yOLED = int(landMarks[8].y*hOLED)

cv2.circle(frame,(x,y),10,(255,0,0), -1)

draw.ellipse((xOLED-3,yOLED-4,xOLED+3,yOLED+4),fill=255)

cnt = 0

for pos in colorDot:

if abs(x-pos[0])<radius and abs(y-pos[1])<radius:

colorActive=colors[cnt]

colorLabel =oledText[cnt]

#draw.text((int(wOLED/2),int(hOLED/2)),oledText[cnt],fill=255)

strip.fill((colorActive[2],colorActive[1],colorActive[0]))

strip.show()

cnt=cnt+1

draw.text((int(wOLED/2),int(hOLED/2)),colorLabel,fill=255)

oled.image(image)

oled.show()

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

This is the core circuit we use for the class:

Fusion Hat Circuit Diagram — This is the circuit we will use moving forward in the class

This is the schematic to add the OLED:

Then this is the schematic to add the NeoPixel

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 49: Live Animated Avatars on the SSD1306 OLED With OpenCV

September 10, 2026 admin

Hey everybody! Welcome back to the AI on the Edge series!

In Lesson 49, we’re doing something that’s just plain cool — we’re turning that tiny little SSD1306 OLED into a live animated avatar that follows your face and hands in real time!

That’s right — no more static images, and no more being tethered to your desktop and monitor. This time your OLED is alive! Move your head, smile, raise your eyebrows, wave your hands… and the little display instantly mirrors your movements like a miniature digital twin. It’s like having a tiny version of yourself living on that 128×64 screen!

What You’ll Learn in This Lesson:

How to use MediaPipe Face Mesh to track 468 facial landmarks
How to combine it with Hand Tracking at the same time
The trick to projecting those landmarks onto a tiny OLED in real time
How to draw both points and connecting lines to create smooth, animated avatars
How to keep everything running fast on the Raspberry Pi

This is one of those projects that makes people stop and say “Wait… how is that even possible on that little screen?!”

By the end of this lesson, you’ll have a foundation for building all kinds of fun real-time avatar projects — everything from simple face mirrors to gesture-controlled characters and beyond.

Why This Is Awesome:

We’re not just displaying video on the big screen anymore — we’re compressing all that AI power down onto a cheap little OLED. This is real AI on the Edge stuff, and it looks incredibly impressive for how simple the hardware is.

Whether you want to build interactive displays, robot faces, wearable tech, or just blow your friends’ minds, this lesson gives you the core technique to make it happen.

Ready to make your OLED come alive?

Fire up the code, run it, and watch your tiny digital self start moving with you. Then start experimenting! Try drawing just the eyes and mouth, or just the hands, or maybe even turn it into a little stick figure avatar.

As always, I want to see what you create with this! Drop your versions in the comments or tag me — I love seeing the creative stuff you guys come up with. Don’t just copy and paste my code. Show you really understand the video by making the code yout own.

Enough talk, lets get this party started!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================

import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import numpy as np
import board
import adafruit_ssd1306
from PIL import Image, ImageDraw, ImageFont
import time

wOLED = 128
hOLED = 64
I2C_ADDRESS = 0x3C
i2c = board.I2C()
oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

W=1280
H=720
tStart = time.time()
fps = 0

piCam = Picamera2(1)
piCam.preview_configuration.main.size = (W, H)
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate = 60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

faceMesh = mp.solutions.face_mesh.FaceMesh(
    
    max_num_faces=1,
    refine_landmarks=True,
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5)

hands = mp.solutions.hands.Hands(
    model_complexity = 0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands=2 )

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)
while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame = piCam.capture_array()
    frame = cv2.flip(frame, -1)

    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    faceResults = faceMesh.process(rgb)
    handResults = hands.process(rgb)
    
    image = Image.new("1",(wOLED,hOLED))
    draw = ImageDraw.Draw(image)
    
    if handResults.multi_hand_landmarks:
        for myHands in handResults.multi_hand_landmarks:
            landMarks = myHands.landmark
            lmAll =[]
            for lm in landMarks:
                x = int(lm.x*W)
                y = int(lm.y*H)
                cv2.circle(frame,(x,y),10,(255,0,0),-1)
                xOLED = int(lm.x*wOLED)
                yOLED = int(lm.y*hOLED)
                draw.point((xOLED,yOLED), fill =255)
                lmAll.append((xOLED,yOLED))
            #Uncomment out these lines if you want
            #Lines between the finger joints
#             for idx in mp.solutions.hands.HAND_CONNECTIONS:
#                 lineStart = idx[0]
#                 lineEnd = idx[1]
#                 pt1 = lmAll[lineStart]
#                 pt2 = lmAll[lineEnd]
#                 draw.line((pt1[0],pt1[1],pt2[0],pt2[1]),width=1,fill=255)
#                              
    if faceResults.multi_face_landmarks:
        for faceLandMarks in faceResults.multi_face_landmarks:
            #print(faceLandmarks)
            lmAll = []
            for lm in faceLandMarks.landmark:
                xOLED = int(lm.x*wOLED)
                yOLED = int(lm.y*hOLED)
                draw.point((xOLED,yOLED), fill = 255)
                #print(xOLED,yOLED)
                lmAll.append((xOLED,yOLED))
            for idx in mp.solutions.face_mesh.FACEMESH_CONTOURS:
                #print(idx)
                lineStart = idx[0]
                lineEnd = idx[1]
                pt1 = lmAll[lineStart]
                pt2 = lmAll[lineEnd]
                draw.line((pt1[0],pt1[1],pt2[0],pt2[1]), width=1, fill = 255)



    oled.image(image)
    oled.show()


    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

    cv2.imshow("Camera", frame)

    if cv2.waitKey(1) == ord('q'):
        break

cv2.destroyAllWindows()
piCam.stop()
oled.fill(0)
oled.show()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import numpy as np

import board

import adafruit_ssd1306

from PIL import Image, ImageDraw, ImageFont

import time

wOLED = 128

hOLED = 64

I2C_ADDRESS = 0x3C

i2c = board.I2C()

oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

W=1280

H=720

tStart = time.time()

fps = 0

piCam = Picamera2(1)

piCam.preview_configuration.main.size = (W, H)

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate = 60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

faceMesh = mp.solutions.face_mesh.FaceMesh(

max_num_faces=1,

refine_landmarks=True,

min_detection_confidence=0.5,

min_tracking_confidence=0.5)

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands=2 )

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame = piCam.capture_array()

frame = cv2.flip(frame, -1)

rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

faceResults = faceMesh.process(rgb)

handResults = hands.process(rgb)

image = Image.new("1",(wOLED,hOLED))

draw = ImageDraw.Draw(image)

if handResults.multi_hand_landmarks:

for myHands in handResults.multi_hand_landmarks:

landMarks = myHands.landmark

lmAll =[]

for lm in landMarks:

x = int(lm.x*W)

y = int(lm.y*H)

cv2.circle(frame,(x,y),10,(255,0,0),-1)

xOLED = int(lm.x*wOLED)

yOLED = int(lm.y*hOLED)

draw.point((xOLED,yOLED), fill =255)

lmAll.append((xOLED,yOLED))

#Uncomment out these lines if you want

#Lines between the finger joints

# for idx in mp.solutions.hands.HAND_CONNECTIONS:

# lineStart = idx[0]

# lineEnd = idx[1]

# pt1 = lmAll[lineStart]

# pt2 = lmAll[lineEnd]

# draw.line((pt1[0],pt1[1],pt2[0],pt2[1]),width=1,fill=255)

if faceResults.multi_face_landmarks:

for faceLandMarks in faceResults.multi_face_landmarks:

#print(faceLandmarks)

lmAll = []

for lm in faceLandMarks.landmark:

xOLED = int(lm.x*wOLED)

yOLED = int(lm.y*hOLED)

draw.point((xOLED,yOLED), fill = 255)

#print(xOLED,yOLED)

lmAll.append((xOLED,yOLED))

for idx in mp.solutions.face_mesh.FACEMESH_CONTOURS:

#print(idx)

lineStart = idx[0]

lineEnd = idx[1]

pt1 = lmAll[lineStart]

pt2 = lmAll[lineEnd]

draw.line((pt1[0],pt1[1],pt2[0],pt2[1]), width=1, fill = 255)

oled.image(image)

oled.show()

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

if cv2.waitKey(1) == ord('q'):

break

cv2.destroyAllWindows()

piCam.stop()

oled.fill(0)

oled.show()

Technology Tutorials

Tag Archives: Mediapipe

AI on the Edge LESSON 54: Custom Hand Gesture Recognition & Dynamic NeoPixel Control

System Architecture Overview

Gesture-to-Hardware Animation Mapping

Interactive Keyboard Controls

Hardware Wiring Reference

Homework Challenge

Homework & Next Steps

AI on the Edge LESSON 54: Custom Hand Gesture Recognition and Dynamic NeoPixel Control

What We Are Building Today

Hardware Setup & Wiring (SPI Interface)

Wiring Diagram Guide

⌨️ Interactive Keyboard Controls

Complete Python Source Code

Homework Assignment

AI on the Edge LESSON 53: Gesture Recognition on the Pi 5 with Training File

AI on the Edge LESSON 50: Control NeoPixel Ring With Hand Gestures and MediaPipe

AI on the Edge LESSON 49: Live Animated Avatars on the SSD1306 OLED With OpenCV

What You’ll Learn in This Lesson:

Why This Is Awesome:

Making The World a Better Place One High Tech Project at a Time. Enjoy!