The major challenge we face in this AI on the Edge class is getting a Raspberry Pi 5 configures where you have all the AI Models, Libraries, Modules and Methods installed, and where they all play nicely together. Often, when you add a new model, the old model becomes broken. This is because when you install something new, it often times updates the dependencies. That means it updates a library already on your system. For example, lets say you have numpy 14, working with YOLO 11. Now you install mediapipe, and it updates numpy 14 to numpy 15. This then ‘Breaks’ your YOLO, as it wanted a different version of numpy. Likely you will get frustrated and quit before you get the dependency problems solved. In order to get around this, you can use a special education version of the Bookworm OS, which has all the needed libraries installed already and working nicely with each other. The video above shows you how to install this OS. Once you do, no not update it, do not upgrade it, do not touch it. Use it to develop your programs and projects for this class. If you want to do something else with your pi, have a separate SD card.
Tag Archives: AI
AI on the Edge LESSON 1: Introduction and Class Overview
Welcome to our all new AI on the Edge class! I will need you to buckle up, get your hardware together, and get ready to teach AI who is boss! We will be using a Pi 5, and the Fusion AI Lab kit. I will show links to the hardware below. In today’s lesson I describe the Class Introduction, and will show you some demos of the types of projects we will be doing. You will either Drive AI or your will be Destroyed by AI. Don’t be one of the ones who will be eaten by it
Guys, get your gear, and make sure you end up on the right side of the Dystopian future that awaits the world.
I have provided Amazon links, so you can order everything in the same place
You Will need a Raspberry Pi 5
Order Pi 5
You will need a heat sink and fan
Order Heat Sink and Fan
You Will Need the Fusion AI Lab Kit
Order Fusion AI Lab Kit
You Will Need a 25 Watt Power Supply
Order Power Supply
You Will Need a Micro HDMI Cable
Order Micro HDMI Cable
You Will Need a Keyboard and Mouse
Order Wireless Keyboard and Mouse
This isn’t just another Raspberry Pi class. This is a complete journey where we’re going to take the powerful Raspberry Pi 5, combine it with the SunFounder Fusion AI Lab kit, and build real, practical, intelligent systems that run completely on the edge — no cloud, no internet required.
In this class, you’re not going to just learn how to blink an LED or run someone else’s pre-made script. You’re going to learn how to build smart machines that can see, listen, speak, think, and act in the real world. We’re going to combine computer vision, voice recognition, speech synthesis, sensor reading, motor control, and modern AI techniques — all running locally on your Raspberry Pi 5.
Over the course of this series, you will learn how to:
- Capture and process live video from the Raspberry Pi Camera
- Detect faces and track objects in real time using MediaPipe and OpenCV
- Control hardware with voice commands
- Make your Raspberry Pi speak with natural-sounding Text-to-Speech
- Build smooth, responsive control systems using threading
- Use displays like the SSD1306 OLED to show live information
- Combine everything into impressive AI-powered projects
This class is designed for makers, students, hobbyists, and engineers who want to move beyond basic tutorials and start building real intelligent edge devices. Whether you dream of building smart robots, autonomous monitoring systems, interactive AI companions, or just want to gain serious skills in modern embedded AI, this class is for you.
I’m going to teach this the way I always do — step by step, clearly, and with lots of hands-on projects. We’ll start with the fundamentals and gradually build up to more advanced and exciting projects as the class progresses.
If you’ve ever wanted to move from “playing with the Raspberry Pi” to “building truly intelligent systems,” then you’re in the right place. This is going to be a fun, challenging, and incredibly rewarding journey.
So if you’re ready to stop just watching AI videos and start building your own AI on the edge… then buckle up, because we’re about to do exactly that.
Welcome to the class! I’m really glad you’re here. Let’s get started!
Object Detection Using YOLO and RTSP Camera on Raspberry Pi 5
OK guys, you spoke, and I listened. You all are asking for a lesson on how to do object detection on a Pi 5 using YOLO and an IP Camera. Well, you are about to get what you asked for. We will make this work, or we will DIE TRYING. Never fear, once you watch the video you will both understand and be able to do it on your own. First, I am assuming you watched our previous lesson where I showed you how to do the basic install and setup of YOLO. If not, never fear, I have the commands below. NOTE: This tutorial is geared towards bookworm OS. I strongly suggest you start with a fresh bookworm SC card, as there are many dependencies, and it is most likely to work if you start exactly where I am starting . . . with a fresh OS. Thes these are the commands I shared last week to get YOLO up and working: (just open a terminal, and paste these commands one at a time)
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
# 1. Configure X11 (manual steps required) sudo raspi-config # → Go to Advanced Options → X11 → Enable X11 # → Finish and reboot when prompted # 5. Update system and install OpenCV sudo apt update sudo apt full-upgrade -y sudo apt install python3-opencv -y # Verify OpenCV python3 -c "import cv2; print('OpenCV version:', cv2.__version__)" # Expected output: something like "OpenCV version: 4.6.0" or higher # Create and activate virtual environment for YOLO11 (Ultralytics) python3 -m venv --system-site-packages YOLO source YOLO/bin/activate # You are now inside the (YOLO) virtual environment # Install Ultralytics YOLO11 inside it pip install "numpy<2.0" ultralytics # Now create a Pi friendly YOLO11 model yolo export model=yolo11n.pt format=ncnn # Optional: Verify YOLO installation python -c "from ultralytics import YOLO; print('Ultralytics YOLO ready')" # When finished working with YOLO, you can deactivate with: # deactivate #Now open Thonny, and you need to point thonny to the virtual environment you #just created. Open tools- options, select 'interpreter' tab, then click they Python #executable, selecting ... and navigate from home directory, #to YOLO, to bin, and then select python |
Now, I will explain this code, and will help you configure it for your cameras, but you will need to open up thonny, and paste in the following code as a start. IMPORTANT, as mentioned above, you need to set interpreter in thonny to the virtual environment set up in the process above. If this is not familiar to you, go back and watch last weeks lesson (click previous at the bottom of this post). Without further adue, here is the code we will work with today:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
import cv2 from ultralytics import YOLO #import secret import threading import time W=1280 H=720 #Now, as eplained in the video, edit the line below where #you put your wifi user name in place of user, and your #wifi password in place of password, and you put in your #IP address RTSP_URL = "rtsp://user:password@192.168.88.44:554/cam/realmonitor?channel=1&subtype=0" #RTSP_URL = secret.RTSP_URL1 #I have put my wifi credentials in a secret file so no #one sees my credentials. You will just use the RTSP_URL line above. # Load the exported NCNN model (replace with your model path) model = YOLO("/home/pjm/yolo11n_ncnn_model/", task="detect") lock = threading.Lock() running = True def frameGrabber(url): global ipFrame, running cap = cv2.VideoCapture(url, cv2.CAP_FFMPEG) cap.set(cv2.CAP_PROP_BUFFERSIZE, 1) cap.set(cv2.CAP_PROP_POS_FRAMES, 0) # Reset frame position while running: ret, frame = cap.read() if ret: with lock: #frame = cv2.resize(frame, (W, H)) ipFrame = frame.copy() cap.release() print("Thread Terminated") thread = threading.Thread(target=frameGrabber, args=(RTSP_URL,), daemon=True) thread.start() tStart=time.time() time.sleep(2) fps=0 cnt=0 # Set resolution for faster processing (optional, adjust based on your needs) while True: deltaT=time.time()-tStart fps= fps*.9 + .1/deltaT tStart=time.time() with lock: ipFrameShow=ipFrame.copy() results = model(ipFrameShow, conf=0.25, verbose=False)[0] # conf: confidence threshold; adjust as needed # Annotate the frame with detections (boxes, labels, scores) annotatedFrame = results.plot() annotatedFrame=cv2.resize(annotatedFrame, (W,H)) cv2.putText(annotatedFrame, "FPS: "+str(round(fps,1)), (int(W*.01), int(H*.075)), cv2.FONT_HERSHEY_SIMPLEX, H*.002, (0, 0, 255), 3) cv2.imshow("IP Camera", annotatedFrame) #cv2.moveWindow("IP Camera",100,100) if cv2.waitKey(1)==ord('q'): break running=False thread.join() # Wait for thread to fully exit time.sleep(1) cv2.destroyAllWindows() import gc gc.collect() # Force garbage collection to reclaim memory/connections #Make Extra sure all processes are killed import os os._exit(0) print("Program Terminated") |
The video explains everything, please watch it!
AI for Everyone LESSON 29: Control of Real World Objects with Gesture Recognition in Mediapipe
In this video lesson we show you how you can control objects in the real world using OpenCV, Python, Mediapipa and our old friend, the Arduino. On the Python side, we recognize hand gestures, and then we pass the recognized gesture to Arduino and Arduino lights LED in response to what hand signal is seen. This is a simple example, but a very powerful method. Instead of LED, you could operate servos, stepper motors or relays to control any manner of different devices. For your convenience, this is the code we used on the Arduino side:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 |
String cmd; int LED1=9; int LED2=6; int LED3=5; int LED4=3; int LED5=10; void setup() { pinMode(LED1,OUTPUT); pinMode(LED2,OUTPUT); pinMode(LED3,OUTPUT); pinMode(LED4,OUTPUT); pinMode(LED5,OUTPUT); Serial.begin(115200); } void loop() { while (Serial.available()==0){ } cmd = Serial.readStringUntil('\r'); if (cmd=="One"){ digitalWrite(LED1,HIGH); digitalWrite(LED2,LOW); digitalWrite(LED3,LOW); digitalWrite(LED4,LOW); digitalWrite(LED5,LOW); } if (cmd=="Two"){ digitalWrite(LED1,HIGH); digitalWrite(LED2,HIGH); digitalWrite(LED3,LOW); digitalWrite(LED4,LOW); digitalWrite(LED5,LOW); } if (cmd=="Three"){ digitalWrite(LED1,HIGH); digitalWrite(LED2,HIGH); digitalWrite(LED3,HIGH); digitalWrite(LED4,LOW); digitalWrite(LED5,LOW); } if (cmd=="Four"){ digitalWrite(LED1,HIGH); digitalWrite(LED2,HIGH); digitalWrite(LED3,HIGH); digitalWrite(LED4,HIGH); digitalWrite(LED5,LOW); } if (cmd=="Five"){ digitalWrite(LED1,HIGH); digitalWrite(LED2,HIGH); digitalWrite(LED3,HIGH); digitalWrite(LED4,HIGH); digitalWrite(LED5,HIGH); } if (cmd=="Pinky"){ digitalWrite(LED1,LOW); digitalWrite(LED2,LOW); digitalWrite(LED3,LOW); digitalWrite(LED4,HIGH); digitalWrite(LED5,LOW); } if (cmd=="Thumb"){ digitalWrite(LED1,LOW); digitalWrite(LED2,LOW); digitalWrite(LED3,LOW); digitalWrite(LED4,LOW); digitalWrite(LED5,HIGH); } if (cmd=="Inside"){ digitalWrite(LED1,LOW); digitalWrite(LED2,HIGH); digitalWrite(LED3,HIGH); digitalWrite(LED4,LOW); digitalWrite(LED5,LOW); } if (cmd=="Outside"){ digitalWrite(LED1,HIGH); digitalWrite(LED2,LOW); digitalWrite(LED3,LOW); digitalWrite(LED4,HIGH); digitalWrite(LED5,LOW); } if (cmd=="Unknown"){ digitalWrite(LED1,LOW); digitalWrite(LED2,LOW); digitalWrite(LED3,LOW); digitalWrite(LED4,LOW); digitalWrite(LED5,LOW); } } |
And on the python side, we used the following code.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 |
import time import cv2 print(cv2.__version__) import numpy as np import pickle import serial arduinoData = serial.Serial('COM3',115200) class mpHands: import mediapipe as mp def __init__(self,maxHands=2,tol1=.5,tol2=.5): self.hands=self.mp.solutions.hands.Hands(False,maxHands,tol1,tol2) def Marks(self,frame): myHands=[] frameRGB=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB) results=self.hands.process(frameRGB) if results.multi_hand_landmarks != None: for handLandMarks in results.multi_hand_landmarks: myHand=[] for landMark in handLandMarks.landmark: myHand.append((int(landMark.x*width),int(landMark.y*height))) myHands.append(myHand) return myHands def findDistances(handData): distMatrix=np.zeros([len(handData),len(handData)],dtype='float') palmSize=((handData[0][0]-handData[9][0])**2+(handData[0][1]-handData[9][1])**2)**(1./2.) for row in range(0,len(handData)): for column in range(0,len(handData)): distMatrix[row][column]=(((handData[row][0]-handData[column][0])**2+(handData[row][1]-handData[column][1])**2)**(1./2.))/palmSize return distMatrix def findError(gestureMatrix,unknownMatrix,keyPoints): error=0 for row in keyPoints: for column in keyPoints: error=error+abs(gestureMatrix[row][column]-unknownMatrix[row][column]) print(error) return error def findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol): errorArray=[] for i in range(0,len(gestNames),1): error=findError(knownGestures[i],unknownGesture,keyPoints) errorArray.append(error) errorMin=errorArray[0] minIndex=0 for i in range(0,len(errorArray),1): if errorArray[i]<errorMin: errorMin=errorArray[i] minIndex=i if errorMin<tol: gesture=gestNames[minIndex] if errorMin>=tol: gesture='Unknown' return gesture width=1280 height=720 cam=cv2.VideoCapture(3,cv2.CAP_DSHOW) cam.set(cv2.CAP_PROP_FRAME_WIDTH, width) cam.set(cv2.CAP_PROP_FRAME_HEIGHT,height) cam.set(cv2.CAP_PROP_FPS, 30) cam.set(cv2.CAP_PROP_FOURCC,cv2.VideoWriter_fourcc(*'MJPG')) findHands=mpHands(1) time.sleep(5) keyPoints=[0,4,5,9,13,17,8,12,16,20] train=int(input('Enter 1 to Train, Enter 0 to Recognize ')) if train==1: trainCnt=0 knownGestures=[] numGest=int(input('How Many Gestures Do You Want? ')) gestNames=[] for i in range(0,numGest,1): prompt='Name of Gesture #'+str(i+1)+' ' name=input(prompt) gestNames.append(name) print(gestNames) trainName=input('Filename for training data? (Press Enter for Default) ') if trainName=='': trainName='default' trainName=trainName+'.pkl' if train==0: trainName=input('What Training Data Do You Want to Use? (Press Enter for Default) ') if trainName=='': trainName='default' trainName=trainName+'.pkl' with open(trainName,'rb') as f: gestNames=pickle.load(f) knownGestures=pickle.load(f) tol=10 while True: ignore, frame = cam.read() frame=cv2.resize(frame,(width,height)) handData=findHands.Marks(frame) if train==1: if handData!=[]: print('Please Show Gesture ',gestNames[trainCnt],': Press t when Ready') if cv2.waitKey(1) & 0xff==ord('t'): knownGesture=findDistances(handData[0]) knownGestures.append(knownGesture) trainCnt=trainCnt+1 if trainCnt==numGest: train=0 with open(trainName,'wb') as f: pickle.dump(gestNames,f) pickle.dump(knownGestures,f) if train == 0: if handData!=[]: unknownGesture=findDistances(handData[0]) myGesture=findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol) #error=findError(knownGesture,unknownGesture,keyPoints) cv2.putText(frame,myGesture,(100,175),cv2.FONT_HERSHEY_SIMPLEX,3,(255,0,0),8) myGesture=myGesture+'\r' arduinoData.write(myGesture.encode()) for hand in handData: for ind in keyPoints: cv2.circle(frame,hand[ind],25,(255,0,255),3) cv2.imshow('my WEBcam', frame) cv2.moveWindow('my WEBcam',0,0) if cv2.waitKey(1) & 0xff ==ord('q'): break cam.release() |
Improved Gesture Recognition in Python and MediaPipe
In this video lesson we show you how you can improve the accuracy of your gesture recognition program developed in the last lesson. We do this by normalizing the hand landmarks distance matrix to a standard size. By doing this, you get accurate results independent of the distance your hand is from the camera. For your convenience, I include the code below which we develop in this lesson. Enjoy!
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
import time import cv2 print(cv2.__version__) import numpy as np class mpHands: import mediapipe as mp def __init__(self,maxHands=2,tol1=.5,tol2=.5): self.hands=self.mp.solutions.hands.Hands(False,maxHands,tol1,tol2) def Marks(self,frame): myHands=[] frameRGB=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB) results=self.hands.process(frameRGB) if results.multi_hand_landmarks != None: for handLandMarks in results.multi_hand_landmarks: myHand=[] for landMark in handLandMarks.landmark: myHand.append((int(landMark.x*width),int(landMark.y*height))) myHands.append(myHand) return myHands def findDistances(handData): distMatrix=np.zeros([len(handData),len(handData)],dtype='float') palmSize=((handData[0][0]-handData[9][0])**2+(handData[0][1]-handData[9][1])**2)**(1./2.) for row in range(0,len(handData)): for column in range(0,len(handData)): distMatrix[row][column]=(((handData[row][0]-handData[column][0])**2+(handData[row][1]-handData[column][1])**2)**(1./2.))/palmSize return distMatrix def findError(gestureMatrix,unknownMatrix,keyPoints): error=0 for row in keyPoints: for column in keyPoints: error=error+abs(gestureMatrix[row][column]-unknownMatrix[row][column]) print(error) return error def findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol): errorArray=[] for i in range(0,len(gestNames),1): error=findError(knownGestures[i],unknownGesture,keyPoints) errorArray.append(error) errorMin=errorArray[0] minIndex=0 for i in range(0,len(errorArray),1): if errorArray[i]<errorMin: errorMin=errorArray[i] minIndex=i if errorMin<tol: gesture=gestNames[minIndex] if errorMin>=tol: gesture='Unknown' return gesture width=1280 height=720 cam=cv2.VideoCapture(4,cv2.CAP_DSHOW) cam.set(cv2.CAP_PROP_FRAME_WIDTH, width) cam.set(cv2.CAP_PROP_FRAME_HEIGHT,height) cam.set(cv2.CAP_PROP_FPS, 30) cam.set(cv2.CAP_PROP_FOURCC,cv2.VideoWriter_fourcc(*'MJPG')) findHands=mpHands(1) time.sleep(5) keyPoints=[0,4,5,9,13,17,8,12,16,20] train=True tol=10 trainCnt=0 knownGestures=[] numGest=int(input('How Many Gestures Do You Want? ')) gestNames=[] for i in range(0,numGest,1): prompt='Name of Gesture #'+str(i+1)+' ' name=input(prompt) gestNames.append(name) print(gestNames) while True: ignore, frame = cam.read() frame=cv2.resize(frame,(width,height)) handData=findHands.Marks(frame) if train==True: if handData!=[]: print('Please Show Gesture ',gestNames[trainCnt],': Press t when Ready') if cv2.waitKey(1) & 0xff==ord('t'): knownGesture=findDistances(handData[0]) knownGestures.append(knownGesture) trainCnt=trainCnt+1 if trainCnt==numGest: train=False if train == False: if handData!=[]: unknownGesture=findDistances(handData[0]) myGesture=findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol) #error=findError(knownGesture,unknownGesture,keyPoints) cv2.putText(frame,myGesture,(100,175),cv2.FONT_HERSHEY_SIMPLEX,3,(255,0,0),8) for hand in handData: for ind in keyPoints: cv2.circle(frame,hand[ind],25,(255,0,255),3) cv2.imshow('my WEBcam', frame) cv2.moveWindow('my WEBcam',0,0) if cv2.waitKey(1) & 0xff ==ord('q'): break cam.release() |
s lesson. Enjoy!