Welcome back, everyone! In this lesson, we are taking our custom MediaPipe hand gesture recognition system to the next level. Up until now, every time you stopped your Python script, all of your hard-earned gesture training data vanished into thin air. If you wanted to run your project again, you had to re-train every single hand sign from scratch.
That ends today. We are going to make our AI remember what it learned by saving our gesture feature vectors to a local file using Python’s pickle library.
What You Will Learn in This Lesson
-
Data Persistence: How to serialize and save Python dictionaries containing landmark distance vectors directly to disk (
gesture_data.pkl). -
Automated Setup: How to check for existing training files at boot so your program automatically skips the initial training prompt if data already exists.
-
Dynamic Training Management: How to interactively train new gestures on the fly (
't'key) or delete outdated gestures ('d'key) during runtime while automatically updating the saved file. -
Robust Error Handling: Ensuring index tracking (
trainIdx) safely syncs with live gesture arrays so pressing the spacebar never crashes your video feed.
Key Concepts Explained
1. Pickling Data Objects In MediaPipe gesture recognition, a trained gesture isn’t a complex neural network weight file; it’s simply a dictionary mapping a gesture name (string) to a list of normalized keypoint ratios (floating-point numbers). Python’s pickle module allows us to convert this dictionary into a byte stream and write it directly to a file (.pkl). When the program launches, pickle.load() reconstructs the dictionary in milliseconds.
2. Managing Execution Flow on Startup When gesture_data.pkl exists, our program automatically populates gestureNames with the keys from trainedData and sets trainIdx past the end of the array. This keeps the camera pipeline running in Inference Mode immediately. If no file exists, the script seamlessly drops into Training Setup Mode to prompt the user for initial inputs.
This is the code we develop in this video lesson:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 |
# ==================================================================== # DISCLAIMER: # This code is provided as-is for educational and experimental # purposes only. The author makes no representations or warranties of # any kind concerning the safety, suitability, or accuracy of this # code. Use at your own risk. The author assumes no liability for any # damages, system failures, security breaches, or network issues # resulting from the use or implementation of this script. # ==================================================================== import cv2 import time from picamera2 import Picamera2 import mediapipe as mp import math import os import pickle DATA_FILE = "gesture_data.pkl" def saveTrainedData(filename, data): with open(filename, 'wb') as f: pickle.dump(data,f) print("[INFO] Saved Gesture Data to "+str(filename)) def loadTrainedData(filename): if os.path.exists(filename): with open(filename, 'rb') as f: data = pickle.load(f) print("[INFO] LOADED "+str(len(data))+ " gestures from " + str(filename)) return data if not os.path.exists(filename): return {} def findDistance(pt1, pt2): dX = pt1.x - pt2.x dY = pt1.y - pt2.y dZ = pt1.z - pt2.z dist = math.sqrt(dX*dX + dY*dY + dZ*dZ) return dist def getTipDistances(handLm): lm = handLm.landmark wrist = lm[0] indexBase = lm[5] scaleFactor = findDistance(wrist, indexBase) tips = [lm[4],lm[8],lm[12],lm[16],lm[20]] tipDistances =[] for tip in tips: dist = findDistance( wrist, tip) / scaleFactor tipDistances.append(dist) for startTip in range(len(tips)): for endTip in range(startTip+1, len(tips)): dist = findDistance(tips[startTip],tips[endTip]) / scaleFactor tipDistances.append(dist) return tipDistances def calculateError(tipDist, trainDist): totalError = 0 for idx in range(len(tipDist)): diff = tipDist[idx] -trainDist[idx] totalError = totalError + abs(diff) return totalError def findGesture(tipLive, trainedData, maxAllowedError): bestMatch = "Unknown" lowestError = 999999999 for gestureName, trainedDist in trainedData.items(): error = calculateError(tipLive, trainedDist) if error < lowestError: lowestError = error bestMatch = gestureName if lowestError > maxAllowedError: return "Unknown" if lowestError <= maxAllowedError: return bestMatch trainedData = loadTrainedData(DATA_FILE) gestureNames =[] if not trainedData: print("HAND GESTURE TRAINING SETUP") numGestures = int(input("Enter Number of Gestures to Train On: ")) gestureNames =[] for idx in range(numGestures): name = input("Enter Name for Gesture #"+ str(idx+1)+ ": ") gestureNames.append(name) W=1280 H=720 tStart = time.time() fps = 15 RES = (W,H) piCam = Picamera2(1) piCam.preview_configuration.main.size = RES piCam.preview_configuration.main.format = "RGB888" piCam.preview_configuration.controls.FrameRate=60 piCam.preview_configuration.align() piCam.configure("preview") piCam.start() textLowerLeft = (int(W*.01),int(H*.07)) fontFace = cv2.FONT_HERSHEY_SIMPLEX fontThickness = int(W/425) fontScale = H*.002 fontColor = (0,0,255) textFps = (int(W*.01),int(H*.07)) textGesture= (int(W*.01),int(H*.14)) textGesturePrompt = (int(W*.01),int(H*.21)) maxErrorThreshold = 3.5 hands = mp.solutions.hands.Hands( model_complexity = 0, min_detection_confidence = .5, min_tracking_confidence = .5, max_num_hands =2) trainIdx = 0 while True: deltaT = time.time() - tStart tStart=time.time() fps = fps*.95 + (1/deltaT)*.05 frame= piCam.capture_array() frame=cv2.flip(frame,-1) rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB) results = hands.process(rgbFrame) if results.multi_hand_landmarks: frameHeight, frameWidth, _ = frame.shape for handLm in results.multi_hand_landmarks: # Draw landmark numbers on hand joints for idx, landmark in enumerate(handLm.landmark): pixelX = int(landmark.x * frameWidth) pixelY = int(landmark.y * frameHeight) cv2.putText( frame, str(idx), (pixelX, pixelY), cv2.FONT_HERSHEY_SIMPLEX, .6*fontScale, (255, 0, 0), fontThickness ) if trainIdx < len(gestureNames): targetName = gestureNames[trainIdx] promptText = "show " + targetName + " & Press Space" cv2.putText(frame,promptText, textGesturePrompt, fontFace,fontScale, (0,255,255), fontThickness) detectGesture = "Unknown" if results.multi_hand_landmarks: liveDistances = getTipDistances(results.multi_hand_landmarks[0]) detectGesture = findGesture(liveDistances, trainedData, maxErrorThreshold) labelText = "Gesture: " + detectGesture cv2.putText(frame, labelText, textGesture,fontFace,fontScale, (0,255,0),fontThickness) myText = "FPS: "+str(round(fps,1)) cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness) cv2.imshow("Camera", frame) #cv2.moveWindow("Camera",0,60) key = cv2.waitKey(1) if key == ord('t'): newGestureName = input("Enter Name for New Gesture: ") gestureNames.append(newGestureName) saveTrainedData(DATA_FILE, trainedData) if key == ord('d'): delName = input("Enter Name of Gesture to Remove: ") if delName not in trainedData: print("[WARNING] Gesture "+delName+ " Not Found in Training Data.") if delName in trainedData: del trainedData[delName] saveTrainedData(DATA_FILE,trainedData) print("[INFO] Removed "+delName+"from trained data.") if delName in gestureNames: gestureNames.remove(delName) trainIdx=min(trainIdx,len(gestureNames)) if key == ord(' '): if results.multi_hand_landmarks: targetHand = results.multi_hand_landmarks[0] trainedData[targetName] = getTipDistances(targetHand) trainIdx = trainIdx + 1 if key == ord('q'): break cv2.destroyAllWindows() saveTrainedData(DATA_FILE, trainedData) print('Program Terminated') |
Homework Assignment
Now that your Pi 5 can remember hand gestures across reboots, it’s time to bridge software recognition with real-world hardware control!
The Challenge: Integrate an SPI NeoPixel Ring (WS2812B/SK6812) with your gesture recognition code.
-
Map distinct hand signs to dynamic LED patterns (for example: Fist = Red Pulse, Open Palm = Bright White Beacon, Peace Sign = Dual Spinner, OK Sign = Rainbow Wheel).
-
Make sure your animation updates are smooth and non-blocking so your OpenCV camera frame rate doesn’t drop!
We want to see who in the community can build the most impressive physical light demonstration controlled entirely by hand gestures. Share your video results and code modifications in the comments below!