Tag Archives: AI

AI on the Edge LESSON 1: Introduction and Class Overview

March 29, 2026 admin

Welcome to our all new AI on the Edge class! I will need you to buckle up, get your hardware together, and get ready to teach AI who is boss! We will be using a Pi 5, and the Fusion AI Lab kit. I will show links to the hardware below. In today’s lesson I describe the Class Introduction, and will show you some demos of the types of projects we will be doing. You will either Drive AI or your will be Destroyed by AI. Don’t be one of the ones who will be eaten by it

The Future will Belong to Those Who Can Drive AI

Guys, get your gear, and make sure you end up on the right side of the Dystopian future that awaits the world.

I have provided Amazon links, so you can order everything in the same place

You Will need a Raspberry Pi 5
Order Pi 5

You will need a heat sink and fan
Order Heat Sink and Fan

You Will Need the Fusion AI Lab Kit
Order Fusion AI Lab Kit

You Will Need a 25 Watt Power Supply
Order Power Supply

You Will Need a Micro HDMI Cable
Order Micro HDMI Cable

You Will Need a Keyboard and Mouse
Order Wireless Keyboard and Mouse

OK, get Geared Up with the equipment above, and then next week’s lesson will show you how to configure your pi 5 for this class.

OpenCV, Python, Raspberry Pi

Object Detection Using YOLO and RTSP Camera on Raspberry Pi 5

January 2, 2026 admin

OK guys, you spoke, and I listened. You all are asking for a lesson on how to do object detection on a Pi 5 using YOLO and an IP Camera. Well, you are about to get what you asked for. We will make this work, or we will DIE TRYING. Never fear, once you watch the video you will both understand and be able to do it on your own. First, I am assuming you watched our previous lesson where I showed you how to do the basic install and setup of YOLO. If not, never fear, I have the commands below. NOTE: This tutorial is geared towards bookworm OS. I strongly suggest you start with a fresh bookworm SC card, as there are many dependencies, and it is most likely to work if you start exactly where I am starting . . . with a fresh OS. Thes these are the commands I shared last week to get YOLO up and working: (just open a terminal, and paste these commands one at a time)

# 1. Configure X11 (manual steps required)
sudo raspi-config
# → Go to Advanced Options → X11 → Enable X11
# → Finish and reboot when prompted

# 5. Update system and install OpenCV
sudo apt update
sudo apt full-upgrade -y
sudo apt install python3-opencv -y

# Verify OpenCV
python3 -c "import cv2; print('OpenCV version:', cv2.__version__)"
# Expected output: something like "OpenCV version: 4.6.0" or higher

# Create and activate virtual environment for YOLO11 (Ultralytics)
python3 -m venv --system-site-packages YOLO
source YOLO/bin/activate

# You are now inside the (YOLO) virtual environment
# Install Ultralytics YOLO11 inside it
pip install "numpy<2.0" ultralytics

# Now create a Pi friendly YOLO11 model
yolo export model=yolo11n.pt format=ncnn

# Optional: Verify YOLO installation
python -c "from ultralytics import YOLO; print('Ultralytics YOLO ready')"

# When finished working with YOLO, you can deactivate with:
# deactivate

#Now open Thonny, and you need to point thonny to the virtual environment you 
#just created. Open tools- options, select 'interpreter' tab, then click they Python
#executable, selecting ... and navigate from home directory, 
#to YOLO, to bin, and then select python

# 1. Configure X11 (manual steps required)

sudo raspi-config

# → Go to Advanced Options → X11 → Enable X11

# → Finish and reboot when prompted

# 5. Update system and install OpenCV

sudo apt update

sudo apt full-upgrade -y

sudo apt install python3-opencv -y

# Verify OpenCV

python3 -c "import cv2; print('OpenCV version:', cv2.__version__)"

# Expected output: something like "OpenCV version: 4.6.0" or higher

# Create and activate virtual environment for YOLO11 (Ultralytics)

python3 -m venv --system-site-packages YOLO

source YOLO/bin/activate

# You are now inside the (YOLO) virtual environment

# Install Ultralytics YOLO11 inside it

pip install "numpy<2.0" ultralytics

# Now create a Pi friendly YOLO11 model

yolo export model=yolo11n.pt format=ncnn

# Optional: Verify YOLO installation

python -c "from ultralytics import YOLO; print('Ultralytics YOLO ready')"

# When finished working with YOLO, you can deactivate with:

# deactivate

#Now open Thonny, and you need to point thonny to the virtual environment you

#just created. Open tools- options, select 'interpreter' tab, then click they Python

#executable, selecting ... and navigate from home directory,

#to YOLO, to bin, and then select python

Now, I will explain this code, and will help you configure it for your cameras, but you will need to open up thonny, and paste in the following code as a start. IMPORTANT, as mentioned above, you need to set interpreter in thonny to the virtual environment set up in the process above. If this is not familiar to you, go back and watch last weeks lesson (click previous at the bottom of this post). Without further adue, here is the code we will work with today:

import cv2
from ultralytics import YOLO
#import secret
import threading
import time

W=1280
H=720
#Now, as eplained in the video, edit the line below where 
#you put your wifi user name in place of user, and your 
#wifi password in place of password, and you put in your 
#IP address
RTSP_URL = "rtsp://user:password@192.168.88.44:554/cam/realmonitor?channel=1&subtype=0"
#RTSP_URL = secret.RTSP_URL1
#I have put my wifi credentials in a secret file so no
#one sees my credentials. You will just use the RTSP_URL line above. 
# Load the exported NCNN model (replace with your model path)
model = YOLO("/home/pjm/yolo11n_ncnn_model/", task="detect")
lock = threading.Lock()
running = True
def frameGrabber(url):
    global ipFrame, running
    cap = cv2.VideoCapture(url, cv2.CAP_FFMPEG)
    cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)
    cap.set(cv2.CAP_PROP_POS_FRAMES, 0)  # Reset frame position
    while running:
        ret, frame = cap.read()
        if ret:
            with lock:
                #frame = cv2.resize(frame, (W, H))
                ipFrame = frame.copy()
    cap.release()
    print("Thread Terminated")
thread = threading.Thread(target=frameGrabber, args=(RTSP_URL,), daemon=True)
thread.start()
tStart=time.time()
time.sleep(2)
fps=0
cnt=0
# Set resolution for faster processing (optional, adjust based on your needs)
while True:
    deltaT=time.time()-tStart
    fps= fps*.9 + .1/deltaT
    tStart=time.time()
    with lock:
        ipFrameShow=ipFrame.copy()
    results = model(ipFrameShow, conf=0.25, verbose=False)[0]  # conf: confidence threshold; adjust as needed

    # Annotate the frame with detections (boxes, labels, scores)
    annotatedFrame = results.plot()
    annotatedFrame=cv2.resize(annotatedFrame, (W,H))
    cv2.putText(annotatedFrame, "FPS: "+str(round(fps,1)), (int(W*.01), int(H*.075)), 
            cv2.FONT_HERSHEY_SIMPLEX, H*.002, (0, 0, 255), 3)
    cv2.imshow("IP Camera", annotatedFrame)
    #cv2.moveWindow("IP Camera",100,100)
    if cv2.waitKey(1)==ord('q'):
        break
running=False
thread.join()  # Wait for thread to fully exit
time.sleep(1)
cv2.destroyAllWindows()
import gc
gc.collect()  # Force garbage collection to reclaim memory/connections
#Make Extra sure all processes are killed
import os
os._exit(0)
print("Program Terminated")

import cv2

from ultralytics import YOLO

#import secret

import threading

import time

W=1280

H=720

#Now, as eplained in the video, edit the line below where

#you put your wifi user name in place of user, and your

#wifi password in place of password, and you put in your

#IP address

RTSP_URL = "rtsp://user:password@192.168.88.44:554/cam/realmonitor?channel=1&subtype=0"

#RTSP_URL = secret.RTSP_URL1

#I have put my wifi credentials in a secret file so no

#one sees my credentials. You will just use the RTSP_URL line above.

# Load the exported NCNN model (replace with your model path)

model = YOLO("/home/pjm/yolo11n_ncnn_model/", task="detect")

lock = threading.Lock()

running = True

def frameGrabber(url):

global ipFrame, running

cap = cv2.VideoCapture(url, cv2.CAP_FFMPEG)

cap.set(cv2.CAP_PROP_BUFFERSIZE, 1)

cap.set(cv2.CAP_PROP_POS_FRAMES, 0) # Reset frame position

while running:

ret, frame = cap.read()

if ret:

with lock:

#frame = cv2.resize(frame, (W, H))

ipFrame = frame.copy()

cap.release()

print("Thread Terminated")

thread = threading.Thread(target=frameGrabber, args=(RTSP_URL,), daemon=True)

thread.start()

tStart=time.time()

time.sleep(2)

fps=0

cnt=0

# Set resolution for faster processing (optional, adjust based on your needs)

while True:

deltaT=time.time()-tStart

fps= fps*.9 + .1/deltaT

tStart=time.time()

with lock:

ipFrameShow=ipFrame.copy()

results = model(ipFrameShow, conf=0.25, verbose=False)[0] # conf: confidence threshold; adjust as needed

# Annotate the frame with detections (boxes, labels, scores)

annotatedFrame = results.plot()

annotatedFrame=cv2.resize(annotatedFrame, (W,H))

cv2.putText(annotatedFrame, "FPS: "+str(round(fps,1)), (int(W*.01), int(H*.075)),

cv2.FONT_HERSHEY_SIMPLEX, H*.002, (0, 0, 255), 3)

cv2.imshow("IP Camera", annotatedFrame)

#cv2.moveWindow("IP Camera",100,100)

if cv2.waitKey(1)==ord('q'):

break

running=False

thread.join() # Wait for thread to fully exit

time.sleep(1)

cv2.destroyAllWindows()

import gc

gc.collect() # Force garbage collection to reclaim memory/connections

#Make Extra sure all processes are killed

import os

os._exit(0)

print("Program Terminated")

The video explains everything, please watch it!

Artificial Intelligence, OpenCV

AI for Everyone LESSON 29: Control of Real World Objects with Gesture Recognition in Mediapipe

February 24, 2022 admin

In this video lesson we show you how you can control objects in the real world using OpenCV, Python, Mediapipa and our old friend, the Arduino. On the Python side, we recognize hand gestures, and then we pass the recognized gesture to Arduino and Arduino lights LED in response to what hand signal is seen. This is a simple example, but a very powerful method. Instead of LED, you could operate servos, stepper motors or relays to control any manner of different devices. For your convenience, this is the code we used on the Arduino side:

String cmd;
int LED1=9;
int LED2=6;
int LED3=5;
int LED4=3;
int LED5=10;
void setup() {
  pinMode(LED1,OUTPUT);
  pinMode(LED2,OUTPUT);
  pinMode(LED3,OUTPUT);
   pinMode(LED4,OUTPUT);
   pinMode(LED5,OUTPUT);
 Serial.begin(115200);

}
void loop() {
 while (Serial.available()==0){
 
 }
 cmd = Serial.readStringUntil('\r');
if (cmd=="One"){
  digitalWrite(LED1,HIGH);
  digitalWrite(LED2,LOW);
  digitalWrite(LED3,LOW);
  digitalWrite(LED4,LOW);
  digitalWrite(LED5,LOW);
}
if (cmd=="Two"){
  digitalWrite(LED1,HIGH);
  digitalWrite(LED2,HIGH);
  digitalWrite(LED3,LOW);
  digitalWrite(LED4,LOW);
  digitalWrite(LED5,LOW);
}
 if (cmd=="Three"){
  digitalWrite(LED1,HIGH);
  digitalWrite(LED2,HIGH);
  digitalWrite(LED3,HIGH);
  digitalWrite(LED4,LOW);
  digitalWrite(LED5,LOW);
 }
 if (cmd=="Four"){
  digitalWrite(LED1,HIGH);
  digitalWrite(LED2,HIGH);
  digitalWrite(LED3,HIGH);
  digitalWrite(LED4,HIGH);
  digitalWrite(LED5,LOW);
 }
 if (cmd=="Five"){
  digitalWrite(LED1,HIGH);
  digitalWrite(LED2,HIGH);
  digitalWrite(LED3,HIGH);
  digitalWrite(LED4,HIGH);
  digitalWrite(LED5,HIGH);

 }

 if (cmd=="Pinky"){
  digitalWrite(LED1,LOW);
  digitalWrite(LED2,LOW);
  digitalWrite(LED3,LOW);
  digitalWrite(LED4,HIGH);
  digitalWrite(LED5,LOW);

 }
  if (cmd=="Thumb"){
  digitalWrite(LED1,LOW);
  digitalWrite(LED2,LOW);
  digitalWrite(LED3,LOW);
  digitalWrite(LED4,LOW);
  digitalWrite(LED5,HIGH);

 }
  if (cmd=="Inside"){
  digitalWrite(LED1,LOW);
  digitalWrite(LED2,HIGH);
  digitalWrite(LED3,HIGH);
  digitalWrite(LED4,LOW);
  digitalWrite(LED5,LOW);

 }
   if (cmd=="Outside"){
  digitalWrite(LED1,HIGH);
  digitalWrite(LED2,LOW);
  digitalWrite(LED3,LOW);
  digitalWrite(LED4,HIGH);
  digitalWrite(LED5,LOW);

 }
    if (cmd=="Unknown"){
  digitalWrite(LED1,LOW);
  digitalWrite(LED2,LOW);
  digitalWrite(LED3,LOW);
  digitalWrite(LED4,LOW);
  digitalWrite(LED5,LOW);

 }
}

String cmd;

int LED1=9;

int LED2=6;

int LED3=5;

int LED4=3;

int LED5=10;

void setup() {

pinMode(LED1,OUTPUT);

pinMode(LED2,OUTPUT);

pinMode(LED3,OUTPUT);

pinMode(LED4,OUTPUT);

pinMode(LED5,OUTPUT);

Serial.begin(115200);

}

void loop() {

while (Serial.available()==0){

}

cmd = Serial.readStringUntil('\r');

if (cmd=="One"){

digitalWrite(LED1,HIGH);

digitalWrite(LED2,LOW);

digitalWrite(LED3,LOW);

digitalWrite(LED4,LOW);

digitalWrite(LED5,LOW);

}

if (cmd=="Two"){

digitalWrite(LED1,HIGH);

digitalWrite(LED2,HIGH);

digitalWrite(LED3,LOW);

digitalWrite(LED4,LOW);

digitalWrite(LED5,LOW);

}

if (cmd=="Three"){

digitalWrite(LED1,HIGH);

digitalWrite(LED2,HIGH);

digitalWrite(LED3,HIGH);

digitalWrite(LED4,LOW);

digitalWrite(LED5,LOW);

}

if (cmd=="Four"){

digitalWrite(LED1,HIGH);

digitalWrite(LED2,HIGH);

digitalWrite(LED3,HIGH);

digitalWrite(LED4,HIGH);

digitalWrite(LED5,LOW);

}

if (cmd=="Five"){

digitalWrite(LED1,HIGH);

digitalWrite(LED2,HIGH);

digitalWrite(LED3,HIGH);

digitalWrite(LED4,HIGH);

digitalWrite(LED5,HIGH);

}

if (cmd=="Pinky"){

digitalWrite(LED1,LOW);

digitalWrite(LED2,LOW);

digitalWrite(LED3,LOW);

digitalWrite(LED4,HIGH);

digitalWrite(LED5,LOW);

}

if (cmd=="Thumb"){

digitalWrite(LED1,LOW);

digitalWrite(LED2,LOW);

digitalWrite(LED3,LOW);

digitalWrite(LED4,LOW);

digitalWrite(LED5,HIGH);

}

if (cmd=="Inside"){

digitalWrite(LED1,LOW);

digitalWrite(LED2,HIGH);

digitalWrite(LED3,HIGH);

digitalWrite(LED4,LOW);

digitalWrite(LED5,LOW);

}

if (cmd=="Outside"){

digitalWrite(LED1,HIGH);

digitalWrite(LED2,LOW);

digitalWrite(LED3,LOW);

digitalWrite(LED4,HIGH);

digitalWrite(LED5,LOW);

}

if (cmd=="Unknown"){

digitalWrite(LED1,LOW);

digitalWrite(LED2,LOW);

digitalWrite(LED3,LOW);

digitalWrite(LED4,LOW);

digitalWrite(LED5,LOW);

}

And on the python side, we used the following code.

import time
import cv2
print(cv2.__version__)
import numpy as np
import pickle
import serial

arduinoData = serial.Serial('COM3',115200)

class mpHands:
    import mediapipe as mp
    def __init__(self,maxHands=2,tol1=.5,tol2=.5):
        self.hands=self.mp.solutions.hands.Hands(False,maxHands,tol1,tol2)
    def Marks(self,frame):
        myHands=[]
        frameRGB=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
        results=self.hands.process(frameRGB)
        if results.multi_hand_landmarks != None:
            for handLandMarks in results.multi_hand_landmarks:
                myHand=[]
                for landMark in handLandMarks.landmark:
                    myHand.append((int(landMark.x*width),int(landMark.y*height)))
                myHands.append(myHand)
        return myHands
def findDistances(handData):
    distMatrix=np.zeros([len(handData),len(handData)],dtype='float')
    palmSize=((handData[0][0]-handData[9][0])**2+(handData[0][1]-handData[9][1])**2)**(1./2.)
    for row in range(0,len(handData)):
        for column in range(0,len(handData)):
            distMatrix[row][column]=(((handData[row][0]-handData[column][0])**2+(handData[row][1]-handData[column][1])**2)**(1./2.))/palmSize
    return distMatrix

def findError(gestureMatrix,unknownMatrix,keyPoints):
    error=0
    for row in keyPoints:
        for column in keyPoints:
            error=error+abs(gestureMatrix[row][column]-unknownMatrix[row][column])
    print(error)
    return error
def findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol):
    errorArray=[]
    for i in range(0,len(gestNames),1):
        error=findError(knownGestures[i],unknownGesture,keyPoints)
        errorArray.append(error)
    errorMin=errorArray[0]
    minIndex=0
    for i in range(0,len(errorArray),1):
        if errorArray[i]<errorMin:
            errorMin=errorArray[i]
            minIndex=i
    if errorMin<tol:
        gesture=gestNames[minIndex]
    if errorMin>=tol:
        gesture='Unknown'
    return gesture

width=1280
height=720
cam=cv2.VideoCapture(3,cv2.CAP_DSHOW)
cam.set(cv2.CAP_PROP_FRAME_WIDTH, width)
cam.set(cv2.CAP_PROP_FRAME_HEIGHT,height)
cam.set(cv2.CAP_PROP_FPS, 30)
cam.set(cv2.CAP_PROP_FOURCC,cv2.VideoWriter_fourcc(*'MJPG'))
findHands=mpHands(1)
time.sleep(5)
keyPoints=[0,4,5,9,13,17,8,12,16,20]

train=int(input('Enter 1 to Train, Enter 0 to Recognize '))
if train==1:
    trainCnt=0
    knownGestures=[]
    numGest=int(input('How Many Gestures Do You Want? '))
    gestNames=[]
    for i in range(0,numGest,1):
        prompt='Name of Gesture #'+str(i+1)+' '
        name=input(prompt)
        gestNames.append(name)
    print(gestNames)
    trainName=input('Filename for training data? (Press Enter for Default) ')
    if trainName=='':
        trainName='default'
    trainName=trainName+'.pkl'
if train==0:
    trainName=input('What Training Data Do You Want to Use? (Press Enter for Default) ')
    if trainName=='':
        trainName='default'
    trainName=trainName+'.pkl'
    with open(trainName,'rb') as f:
        gestNames=pickle.load(f)
        knownGestures=pickle.load(f)

tol=10

while True:
    ignore,  frame = cam.read()
    frame=cv2.resize(frame,(width,height))
    handData=findHands.Marks(frame)
    if train==1:
        if handData!=[]:
            print('Please Show Gesture ',gestNames[trainCnt],': Press t when Ready')
            if cv2.waitKey(1) & 0xff==ord('t'):
                knownGesture=findDistances(handData[0])
                knownGestures.append(knownGesture)
                trainCnt=trainCnt+1
                if trainCnt==numGest:
                    train=0
                    with open(trainName,'wb') as f:
                        pickle.dump(gestNames,f)
                        pickle.dump(knownGestures,f)
    if train == 0:
        if handData!=[]:
            unknownGesture=findDistances(handData[0])
            myGesture=findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol)
            #error=findError(knownGesture,unknownGesture,keyPoints)
            cv2.putText(frame,myGesture,(100,175),cv2.FONT_HERSHEY_SIMPLEX,3,(255,0,0),8)
            myGesture=myGesture+'\r'
            arduinoData.write(myGesture.encode())
    for hand in handData:
        for ind in keyPoints:
            cv2.circle(frame,hand[ind],25,(255,0,255),3)
    cv2.imshow('my WEBcam', frame)
    cv2.moveWindow('my WEBcam',0,0)
    if cv2.waitKey(1) & 0xff ==ord('q'):
        break
cam.release()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

import time

import cv2

print(cv2.__version__)

import numpy as np

import pickle

import serial

arduinoData = serial.Serial('COM3',115200)

class mpHands:

import mediapipe as mp

def __init__(self,maxHands=2,tol1=.5,tol2=.5):

self.hands=self.mp.solutions.hands.Hands(False,maxHands,tol1,tol2)

def Marks(self,frame):

myHands=[]

frameRGB=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

results=self.hands.process(frameRGB)

if results.multi_hand_landmarks != None:

for handLandMarks in results.multi_hand_landmarks:

myHand=[]

for landMark in handLandMarks.landmark:

myHand.append((int(landMark.x*width),int(landMark.y*height)))

myHands.append(myHand)

return myHands

def findDistances(handData):

distMatrix=np.zeros([len(handData),len(handData)],dtype='float')

palmSize=((handData[0][0]-handData[9][0])**2+(handData[0][1]-handData[9][1])**2)**(1./2.)

for row in range(0,len(handData)):

for column in range(0,len(handData)):

distMatrix[row][column]=(((handData[row][0]-handData[column][0])**2+(handData[row][1]-handData[column][1])**2)**(1./2.))/palmSize

return distMatrix

def findError(gestureMatrix,unknownMatrix,keyPoints):

error=0

for row in keyPoints:

for column in keyPoints:

error=error+abs(gestureMatrix[row][column]-unknownMatrix[row][column])

print(error)

return error

def findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol):

errorArray=[]

for i in range(0,len(gestNames),1):

error=findError(knownGestures[i],unknownGesture,keyPoints)

errorArray.append(error)

errorMin=errorArray[0]

minIndex=0

for i in range(0,len(errorArray),1):

if errorArray[i]<errorMin:

errorMin=errorArray[i]

minIndex=i

if errorMin<tol:

gesture=gestNames[minIndex]

if errorMin>=tol:

gesture='Unknown'

return gesture

width=1280

height=720

cam=cv2.VideoCapture(3,cv2.CAP_DSHOW)

cam.set(cv2.CAP_PROP_FRAME_WIDTH, width)

cam.set(cv2.CAP_PROP_FRAME_HEIGHT,height)

cam.set(cv2.CAP_PROP_FPS, 30)

cam.set(cv2.CAP_PROP_FOURCC,cv2.VideoWriter_fourcc(*'MJPG'))

findHands=mpHands(1)

time.sleep(5)

keyPoints=[0,4,5,9,13,17,8,12,16,20]

train=int(input('Enter 1 to Train, Enter 0 to Recognize '))

if train==1:

trainCnt=0

knownGestures=[]

numGest=int(input('How Many Gestures Do You Want? '))

gestNames=[]

for i in range(0,numGest,1):

prompt='Name of Gesture #'+str(i+1)+' '

name=input(prompt)

gestNames.append(name)

print(gestNames)

trainName=input('Filename for training data? (Press Enter for Default) ')

if trainName=='':

trainName='default'

trainName=trainName+'.pkl'

if train==0:

trainName=input('What Training Data Do You Want to Use? (Press Enter for Default) ')

if trainName=='':

trainName='default'

trainName=trainName+'.pkl'

with open(trainName,'rb') as f:

gestNames=pickle.load(f)

knownGestures=pickle.load(f)

tol=10

while True:

ignore, frame = cam.read()

frame=cv2.resize(frame,(width,height))

handData=findHands.Marks(frame)

if train==1:

if handData!=[]:

print('Please Show Gesture ',gestNames[trainCnt],': Press t when Ready')

if cv2.waitKey(1) & 0xff==ord('t'):

knownGesture=findDistances(handData[0])

knownGestures.append(knownGesture)

trainCnt=trainCnt+1

if trainCnt==numGest:

train=0

with open(trainName,'wb') as f:

pickle.dump(gestNames,f)

pickle.dump(knownGestures,f)

if train == 0:

if handData!=[]:

unknownGesture=findDistances(handData[0])

myGesture=findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol)

#error=findError(knownGesture,unknownGesture,keyPoints)

cv2.putText(frame,myGesture,(100,175),cv2.FONT_HERSHEY_SIMPLEX,3,(255,0,0),8)

myGesture=myGesture+'\r'

arduinoData.write(myGesture.encode())

for hand in handData:

for ind in keyPoints:

cv2.circle(frame,hand[ind],25,(255,0,255),3)

cv2.imshow('my WEBcam', frame)

cv2.moveWindow('my WEBcam',0,0)

if cv2.waitKey(1) & 0xff ==ord('q'):

break

cam.release()

Artificial Intelligence, OpenCV, Python, Tutorial

Improved Gesture Recognition in Python and MediaPipe

February 10, 2022 admin

In this video lesson we show you how you can improve the accuracy of your gesture recognition program developed in the last lesson. We do this by normalizing the hand landmarks distance matrix to a standard size. By doing this, you get accurate results independent of the distance your hand is from the camera. For your convenience, I include the code below which we develop in this lesson. Enjoy!

import time
import cv2
print(cv2.__version__)
import numpy as np

class mpHands:
    import mediapipe as mp
    def __init__(self,maxHands=2,tol1=.5,tol2=.5):
        self.hands=self.mp.solutions.hands.Hands(False,maxHands,tol1,tol2)
    def Marks(self,frame):
        myHands=[]
        frameRGB=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
        results=self.hands.process(frameRGB)
        if results.multi_hand_landmarks != None:
            for handLandMarks in results.multi_hand_landmarks:
                myHand=[]
                for landMark in handLandMarks.landmark:
                    myHand.append((int(landMark.x*width),int(landMark.y*height)))
                myHands.append(myHand)
        return myHands
def findDistances(handData):
    distMatrix=np.zeros([len(handData),len(handData)],dtype='float')
    palmSize=((handData[0][0]-handData[9][0])**2+(handData[0][1]-handData[9][1])**2)**(1./2.)
    for row in range(0,len(handData)):
        for column in range(0,len(handData)):
            distMatrix[row][column]=(((handData[row][0]-handData[column][0])**2+(handData[row][1]-handData[column][1])**2)**(1./2.))/palmSize
    return distMatrix

def findError(gestureMatrix,unknownMatrix,keyPoints):
    error=0
    for row in keyPoints:
        for column in keyPoints:
            error=error+abs(gestureMatrix[row][column]-unknownMatrix[row][column])
    print(error)
    return error
def findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol):
    errorArray=[]
    for i in range(0,len(gestNames),1):
        error=findError(knownGestures[i],unknownGesture,keyPoints)
        errorArray.append(error)
    errorMin=errorArray[0]
    minIndex=0
    for i in range(0,len(errorArray),1):
        if errorArray[i]<errorMin:
            errorMin=errorArray[i]
            minIndex=i
    if errorMin<tol:
        gesture=gestNames[minIndex]
    if errorMin>=tol:
        gesture='Unknown'
    return gesture


width=1280
height=720
cam=cv2.VideoCapture(4,cv2.CAP_DSHOW)
cam.set(cv2.CAP_PROP_FRAME_WIDTH, width)
cam.set(cv2.CAP_PROP_FRAME_HEIGHT,height)
cam.set(cv2.CAP_PROP_FPS, 30)
cam.set(cv2.CAP_PROP_FOURCC,cv2.VideoWriter_fourcc(*'MJPG'))
findHands=mpHands(1)
time.sleep(5)
keyPoints=[0,4,5,9,13,17,8,12,16,20]
train=True
tol=10
trainCnt=0
knownGestures=[]

numGest=int(input('How Many Gestures Do You Want? '))

gestNames=[]

for i in range(0,numGest,1):
    prompt='Name of Gesture #'+str(i+1)+' '
    name=input(prompt)
    gestNames.append(name)
print(gestNames)

while True:
    ignore,  frame = cam.read()
    frame=cv2.resize(frame,(width,height))
    handData=findHands.Marks(frame)
    if train==True:
        if handData!=[]:
            print('Please Show Gesture ',gestNames[trainCnt],': Press t when Ready')
            if cv2.waitKey(1) & 0xff==ord('t'):
                knownGesture=findDistances(handData[0])
                knownGestures.append(knownGesture)
                trainCnt=trainCnt+1
                if trainCnt==numGest:
                    train=False
    if train == False:
        if handData!=[]:
            unknownGesture=findDistances(handData[0])
            myGesture=findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol)
            #error=findError(knownGesture,unknownGesture,keyPoints)
            cv2.putText(frame,myGesture,(100,175),cv2.FONT_HERSHEY_SIMPLEX,3,(255,0,0),8)
    for hand in handData:
        for ind in keyPoints:
            cv2.circle(frame,hand[ind],25,(255,0,255),3)
    cv2.imshow('my WEBcam', frame)
    cv2.moveWindow('my WEBcam',0,0)
    if cv2.waitKey(1) & 0xff ==ord('q'):
        break
cam.release()

100

101

102

103

104

105

import time

import cv2

print(cv2.__version__)

import numpy as np

class mpHands:

import mediapipe as mp

def __init__(self,maxHands=2,tol1=.5,tol2=.5):

self.hands=self.mp.solutions.hands.Hands(False,maxHands,tol1,tol2)

def Marks(self,frame):

myHands=[]

frameRGB=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

results=self.hands.process(frameRGB)

if results.multi_hand_landmarks != None:

for handLandMarks in results.multi_hand_landmarks:

myHand=[]

for landMark in handLandMarks.landmark:

myHand.append((int(landMark.x*width),int(landMark.y*height)))

myHands.append(myHand)

return myHands

def findDistances(handData):

distMatrix=np.zeros([len(handData),len(handData)],dtype='float')

palmSize=((handData[0][0]-handData[9][0])**2+(handData[0][1]-handData[9][1])**2)**(1./2.)

for row in range(0,len(handData)):

for column in range(0,len(handData)):

distMatrix[row][column]=(((handData[row][0]-handData[column][0])**2+(handData[row][1]-handData[column][1])**2)**(1./2.))/palmSize

return distMatrix

def findError(gestureMatrix,unknownMatrix,keyPoints):

error=0

for row in keyPoints:

for column in keyPoints:

error=error+abs(gestureMatrix[row][column]-unknownMatrix[row][column])

print(error)

return error

def findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol):

errorArray=[]

for i in range(0,len(gestNames),1):

error=findError(knownGestures[i],unknownGesture,keyPoints)

errorArray.append(error)

errorMin=errorArray[0]

minIndex=0

for i in range(0,len(errorArray),1):

if errorArray[i]<errorMin:

errorMin=errorArray[i]

minIndex=i

if errorMin<tol:

gesture=gestNames[minIndex]

if errorMin>=tol:

gesture='Unknown'

return gesture

width=1280

height=720

cam=cv2.VideoCapture(4,cv2.CAP_DSHOW)

cam.set(cv2.CAP_PROP_FRAME_WIDTH, width)

cam.set(cv2.CAP_PROP_FRAME_HEIGHT,height)

cam.set(cv2.CAP_PROP_FPS, 30)

cam.set(cv2.CAP_PROP_FOURCC,cv2.VideoWriter_fourcc(*'MJPG'))

findHands=mpHands(1)

time.sleep(5)

keyPoints=[0,4,5,9,13,17,8,12,16,20]

train=True

tol=10

trainCnt=0

knownGestures=[]

numGest=int(input('How Many Gestures Do You Want? '))

gestNames=[]

for i in range(0,numGest,1):

prompt='Name of Gesture #'+str(i+1)+' '

name=input(prompt)

gestNames.append(name)

print(gestNames)

while True:

ignore, frame = cam.read()

frame=cv2.resize(frame,(width,height))

handData=findHands.Marks(frame)

if train==True:

if handData!=[]:

print('Please Show Gesture ',gestNames[trainCnt],': Press t when Ready')

if cv2.waitKey(1) & 0xff==ord('t'):

knownGesture=findDistances(handData[0])

knownGestures.append(knownGesture)

trainCnt=trainCnt+1

if trainCnt==numGest:

train=False

if train == False:

if handData!=[]:

unknownGesture=findDistances(handData[0])

myGesture=findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol)

#error=findError(knownGesture,unknownGesture,keyPoints)

cv2.putText(frame,myGesture,(100,175),cv2.FONT_HERSHEY_SIMPLEX,3,(255,0,0),8)

for hand in handData:

for ind in keyPoints:

cv2.circle(frame,hand[ind],25,(255,0,255),3)

cv2.imshow('my WEBcam', frame)

cv2.moveWindow('my WEBcam',0,0)

if cv2.waitKey(1) & 0xff ==ord('q'):

break

cam.release()

s lesson. Enjoy!

Artificial Intelligence, Python

AI for Everyone LESSON 26: Accurate Gesture Recognition using Python and MediaPipe

February 3, 2022 admin

In this lesson we demonstrate how to use mediaPipe and Python to create an AI system that can accurately recognize hand gestures. This is follow on work to what we developed in Lesson 25. The code we developed is presented below.

import time
import cv2
print(cv2.__version__)
import numpy as np

class mpHands:
    import mediapipe as mp
    def __init__(self,maxHands=2,tol1=.5,tol2=.5):
        self.hands=self.mp.solutions.hands.Hands(False,maxHands,tol1,tol2)
    def Marks(self,frame):
        myHands=[]
        frameRGB=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
        results=self.hands.process(frameRGB)
        if results.multi_hand_landmarks != None:
            for handLandMarks in results.multi_hand_landmarks:
                myHand=[]
                for landMark in handLandMarks.landmark:
                    myHand.append((int(landMark.x*width),int(landMark.y*height)))
                myHands.append(myHand)
        return myHands
def findDistances(handData):
    distMatrix=np.zeros([len(handData),len(handData)],dtype='float')
    for row in range(0,len(handData)):
        for column in range(0,len(handData)):
            distMatrix[row][column]=((handData[row][0]-handData[column][0])**2+(handData[row][1]-handData[column][1])**2)**(1./2.)
    return distMatrix

def findError(gestureMatrix,unknownMatrix,keyPoints):
    error=0
    for row in keyPoints:
        for column in keyPoints:
            error=error+abs(gestureMatrix[row][column]-unknownMatrix[row][column])
    return error
def findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol):
    errorArray=[]
    for i in range(0,len(gestNames),1):
        error=findError(knownGestures[i],unknownGesture,keyPoints)
        errorArray.append(error)
    errorMin=errorArray[0]
    minIndex=0
    for i in range(0,len(errorArray),1):
        if errorArray[i]<errorMin:
            errorMin=errorArray[i]
            minIndex=i
    if errorMin<tol:
        gesture=gestNames[minIndex]
    if errorMin>=tol:
        gesture='Unknown'
    return gesture


width=1280
height=720
cam=cv2.VideoCapture(0,cv2.CAP_DSHOW)
cam.set(cv2.CAP_PROP_FRAME_WIDTH, width)
cam.set(cv2.CAP_PROP_FRAME_HEIGHT,height)
cam.set(cv2.CAP_PROP_FPS, 30)
cam.set(cv2.CAP_PROP_FOURCC,cv2.VideoWriter_fourcc(*'MJPG'))
findHands=mpHands(1)
time.sleep(5)
keyPoints=[0,4,5,9,13,17,8,12,16,20]
train=True
tol=1500
trainCnt=0
knownGestures=[]

numGest=int(input('How Many Gestures Do You Want? '))

gestNames=[]

for i in range(0,numGest,1):
    prompt='Name of Gesture #'+str(i+1)+' '
    name=input(prompt)
    gestNames.append(name)
print(gestNames)

while True:
    ignore,  frame = cam.read()
    frame=cv2.resize(frame,(width,height))
    handData=findHands.Marks(frame)
    if train==True:
        if handData!=[]:
            print('Please Show Gesture ',gestNames[trainCnt],': Press t when Ready')
            if cv2.waitKey(1) & 0xff==ord('t'):
                knownGesture=findDistances(handData[0])
                knownGestures.append(knownGesture)
                trainCnt=trainCnt+1
                if trainCnt==numGest:
                    train=False
    if train == False:
        if handData!=[]:
            unknownGesture=findDistances(handData[0])
            myGesture=findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol)
            #error=findError(knownGesture,unknownGesture,keyPoints)
            cv2.putText(frame,myGesture,(100,175),cv2.FONT_HERSHEY_SIMPLEX,3,(255,0,0),8)
    for hand in handData:
        for ind in keyPoints:
            cv2.circle(frame,hand[ind],25,(255,0,255),3)
    cv2.imshow('my WEBcam', frame)
    cv2.moveWindow('my WEBcam',0,0)
    if cv2.waitKey(1) & 0xff ==ord('q'):
        break
cam.release()

100

101

102

103

import time

import cv2

print(cv2.__version__)

import numpy as np

class mpHands:

import mediapipe as mp

def __init__(self,maxHands=2,tol1=.5,tol2=.5):

self.hands=self.mp.solutions.hands.Hands(False,maxHands,tol1,tol2)

def Marks(self,frame):

myHands=[]

frameRGB=cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

results=self.hands.process(frameRGB)

if results.multi_hand_landmarks != None:

for handLandMarks in results.multi_hand_landmarks:

myHand=[]

for landMark in handLandMarks.landmark:

myHand.append((int(landMark.x*width),int(landMark.y*height)))

myHands.append(myHand)

return myHands

def findDistances(handData):

distMatrix=np.zeros([len(handData),len(handData)],dtype='float')

for row in range(0,len(handData)):

for column in range(0,len(handData)):

distMatrix[row][column]=((handData[row][0]-handData[column][0])**2+(handData[row][1]-handData[column][1])**2)**(1./2.)

return distMatrix

def findError(gestureMatrix,unknownMatrix,keyPoints):

error=0

for row in keyPoints:

for column in keyPoints:

error=error+abs(gestureMatrix[row][column]-unknownMatrix[row][column])

return error

def findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol):

errorArray=[]

for i in range(0,len(gestNames),1):

error=findError(knownGestures[i],unknownGesture,keyPoints)

errorArray.append(error)

errorMin=errorArray[0]

minIndex=0

for i in range(0,len(errorArray),1):

if errorArray[i]<errorMin:

errorMin=errorArray[i]

minIndex=i

if errorMin<tol:

gesture=gestNames[minIndex]

if errorMin>=tol:

gesture='Unknown'

return gesture

width=1280

height=720

cam=cv2.VideoCapture(0,cv2.CAP_DSHOW)

cam.set(cv2.CAP_PROP_FRAME_WIDTH, width)

cam.set(cv2.CAP_PROP_FRAME_HEIGHT,height)

cam.set(cv2.CAP_PROP_FPS, 30)

cam.set(cv2.CAP_PROP_FOURCC,cv2.VideoWriter_fourcc(*'MJPG'))

findHands=mpHands(1)

time.sleep(5)

keyPoints=[0,4,5,9,13,17,8,12,16,20]

train=True

tol=1500

trainCnt=0

knownGestures=[]

numGest=int(input('How Many Gestures Do You Want? '))

gestNames=[]

for i in range(0,numGest,1):

prompt='Name of Gesture #'+str(i+1)+' '

name=input(prompt)

gestNames.append(name)

print(gestNames)

while True:

ignore, frame = cam.read()

frame=cv2.resize(frame,(width,height))

handData=findHands.Marks(frame)

if train==True:

if handData!=[]:

print('Please Show Gesture ',gestNames[trainCnt],': Press t when Ready')

if cv2.waitKey(1) & 0xff==ord('t'):

knownGesture=findDistances(handData[0])

knownGestures.append(knownGesture)

trainCnt=trainCnt+1

if trainCnt==numGest:

train=False

if train == False:

if handData!=[]:

unknownGesture=findDistances(handData[0])

myGesture=findGesture(unknownGesture,knownGestures,keyPoints,gestNames,tol)

#error=findError(knownGesture,unknownGesture,keyPoints)

cv2.putText(frame,myGesture,(100,175),cv2.FONT_HERSHEY_SIMPLEX,3,(255,0,0),8)

for hand in handData:

for ind in keyPoints:

cv2.circle(frame,hand[ind],25,(255,0,255),3)

cv2.imshow('my WEBcam', frame)

cv2.moveWindow('my WEBcam',0,0)

if cv2.waitKey(1) & 0xff ==ord('q'):

break

cam.release()

Technology Tutorials

Tag Archives: AI

AI on the Edge LESSON 1: Introduction and Class Overview

Object Detection Using YOLO and RTSP Camera on Raspberry Pi 5

AI for Everyone LESSON 29: Control of Real World Objects with Gesture Recognition in Mediapipe

Improved Gesture Recognition in Python and MediaPipe

AI for Everyone LESSON 26: Accurate Gesture Recognition using Python and MediaPipe

Making The World a Better Place One High Tech Project at a Time. Enjoy!