Artifiial intellignce | Technology Tutorials

Welcome back, my friends! This is Paul McWhorter from toptechboy.com, and welcome to LESSON 57 of our AI on the Edge tutorial series.

In this lesson, we take what we’ve learned about local object detection using MediaPipe and put it to work in the physical world! We are building an active target tracking system using a 2-DOF pan-tilt servo rig driven by the Raspberry Pi and the Fusion HAT.

By calculating the positional error between the center of our targeted object (in this case, a bottle) and the center of our frame, we compute feedback adjustments to continuously drive our pan and tilt servos—keeping the target locked right in the center of the camera view.

What You Will Learn in Lesson 57

Setting Up MediaPipe Object Detection: Configuring the efficientdet_lite0.tflite model with threshold options and maximum result limits.
Integrating Picamera2 with OpenCV: Capturing high-resolution camera feeds (1280 by 720 at 60 FPS) and converting raw frames to MediaPipe-compatible image formats.
Bounding Box Analysis & Centroid Calculation: Extracting bounding box dimensions and computing the precise pixel center of detected targets.
Proportional Error Servo Control: Using gain coefficients to convert pixel displacements into physical angle corrections for pan and tilt servos.
Deadband Implementation: Preventing servo chatter and hunting by defining a deadband region around the center of the screen.

Hardware Requirements

Raspberry Pi 5 (or Raspberry Pi 4)
Raspberry Pi Camera Module (configured with Picamera2)
SunFounder Fusion HAT (or compatible servo controller HAT)
2-DOF Pan-Tilt Servo Kit (Pan attached to Pin 2, Tilt attached to Pin 3)
Target Object (e.g., a water bottle or any object recognized by COCO dataset categories)

How the Proportional Tracking Control Works

To keep the target centered in the video frame, we continuously measure the distance in pixels between the frame’s center and the center of the target’s bounding box:

errorX = boxCenterX – screenCenterX

errorY = boxCenterY – screenCenterY

If the magnitude of the error exceeds our deadBand limit (15 pixels), we adjust the pan and tilt angles proportionally:

panAngle = panAngle – (errorX * panGain)

tiltAngle = tiltAngle + (errorY * tiltGain)

Pan Limits: Clamped between -90 and +90 degrees.
Tilt Limits: Clamped between -45 and +45 degrees.

Homework Assignment

Now that you have basic single-target tracking running smoothly:

Multi-Target Priority: Modify the script so that if multiple target objects appear on screen, the camera locks onto the object closest to the center of the frame rather than defaulting to the first detected bounding box.
Dynamic Smoothness Filter: Add an exponential moving average or low-pass filter to the bounding box center coordinates to smooth out tracking jitter when the object moves rapidly across the field of view.
Audibly speak the Object of Interst to track. This will require you to incorporate Speech to Text voice recognition.

Get your coffee, fire up your Raspberry Pi, copy the code above into your IDE, and let’s get tracking!

General Artificial Intelligence & Edge Computing Concepts

What is Edge AI?

Edge AI refers to the deployment of artificial intelligence algorithms directly on hardware devices, such as microcontrollers, single-board computers, or edge accelerators, located close to the source of data generation rather than relying on cloud-based processing servers.

Key Advantages of Local Inference:

Low Latency: Processing video feeds directly on an edge device eliminates network transmission delay, enabling real-time feedback loops critical for robotics and physical pan-tilt tracking.
Privacy & Security: Sensitive visual or audio data remains on the local hardware without requiring continuous streaming to third-party cloud infrastructure.
Network Independence: Edge deployments operate reliably in remote or bandwidth-constrained environments without requiring active internet connectivity.
Cost Efficiency: Running localized models eliminates recurring per-API-call cloud infrastructure and compute costs.

Proportional-Integral-Derivative Control Concepts:

Proportional Control: Calculates an output value proportional to the current error magnitude. High proportional gain results in aggressive responses, while low gain yields sluggish tracking.
Deadband: A non-responsive range around the zero-error position where control adjustments are intentionally suspended to prevent motor oscillation, noise amplification, and mechanical wear.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
from fusion_hat.servo import Servo
piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
topBar = 65
windowPadding = 0
cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)
cv2.resizeWindow('Camera', W, H)
tSettle = .1
time.sleep(tSettle)
cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"
scoreThreshold = .35
maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)
options = vision.ObjectDetectorOptions(base_options = baseOptions,
                                       score_threshold=scoreThreshold,
                                       max_results=maxObj)
myDetector = vision.ObjectDetector.create_from_options(options)

targetClass = 'bottle'

panPin = 2
tiltPin = 3
panServo = Servo(panPin)
tiltServo = Servo(tiltPin)
panAngle = 0
tiltAngle = 0
panServo.angle(panAngle)
tiltServo.angle(0)

panGain = .015
tiltGain = .015
deadBand = 15

def updateServo(objectCenterX,objectCenterY, width, height):
    global panAngle, tiltAngle
    tSettle = .01
    screenCenterX = width/2
    screenCenterY = height/2
    
    errorX = objectCenterX - screenCenterX
    errorY = objectCenterY - screenCenterY
    
    if abs(errorX) > deadBand:
        panAngle = panAngle - (errorX*panGain)
        if panAngle > 90:
            panAngle = 90
        if panAngle < -90:
            panAngle = -90
        panServo.angle(panAngle)
    if abs(errorY) > deadBand:
        tiltAngle = tiltAngle + (errorY*tiltGain)
        if tiltAngle > 45:
            tiltAngle = 45
        if tiltAngle < -45:
            tiltAngle = -45
        tiltServo.angle(tiltAngle)
        time.sleep(tSettle)
        
            
while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)
    myDetections = myDetector.detect(myImage)
    targetInZone = False
    #print(myDetections.detections)
    cv2.circle(frame,(int(W/2),int(H/2)),deadBand*2,(0,255,0),2)
    for detection in myDetections.detections:
        if detection.categories:
            category=detection.categories[0]
            if category.category_name == targetClass:
                box = detection.bounding_box
                x = int(box.origin_x)
                y = int(box.origin_y)
                w = int(box.width)
                h = int(box.height)
                boxCenterX = x + int(w/2)
                boxCenterY = y + int(h/2)
                cv2.rectangle(frame,(x,y),(x+w,y+h),
                              (255,0,0),fontThickness)
                labelText = "Target Aquired "+str(round(category.score*100))+'%'
                cv2.circle(frame,(boxCenterX,boxCenterY),6,(0,0,255),-1)
                cv2.putText(frame,labelText,(x,y-10),fontFace,fontScale*.8,(0,255,0),2)
                updateServo(boxCenterX,boxCenterY,W,H)
                                     
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
myDetector.close()
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

from mediapipe.tasks import python

from mediapipe.tasks.python import vision

from fusion_hat.servo import Servo

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

topBar = 65

windowPadding = 0

cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)

cv2.resizeWindow('Camera', W, H)

tSettle = .1

time.sleep(tSettle)

cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"

scoreThreshold = .35

maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)

options = vision.ObjectDetectorOptions(base_options = baseOptions,

score_threshold=scoreThreshold,

max_results=maxObj)

myDetector = vision.ObjectDetector.create_from_options(options)

targetClass = 'bottle'

panPin = 2

tiltPin = 3

panServo = Servo(panPin)

tiltServo = Servo(tiltPin)

panAngle = 0

tiltAngle = 0

panServo.angle(panAngle)

tiltServo.angle(0)

panGain = .015

tiltGain = .015

deadBand = 15

def updateServo(objectCenterX,objectCenterY, width, height):

global panAngle, tiltAngle

tSettle = .01

screenCenterX = width/2

screenCenterY = height/2

errorX = objectCenterX - screenCenterX

errorY = objectCenterY - screenCenterY

if abs(errorX) > deadBand:

panAngle = panAngle - (errorX*panGain)

if panAngle > 90:

panAngle = 90

if panAngle < -90:

panAngle = -90

panServo.angle(panAngle)

if abs(errorY) > deadBand:

tiltAngle = tiltAngle + (errorY*tiltGain)

if tiltAngle > 45:

tiltAngle = 45

if tiltAngle < -45:

tiltAngle = -45

tiltServo.angle(tiltAngle)

time.sleep(tSettle)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)

myDetections = myDetector.detect(myImage)

targetInZone = False

#print(myDetections.detections)

cv2.circle(frame,(int(W/2),int(H/2)),deadBand*2,(0,255,0),2)

for detection in myDetections.detections:

if detection.categories:

category=detection.categories[0]

if category.category_name == targetClass:

box = detection.bounding_box

x = int(box.origin_x)

y = int(box.origin_y)

w = int(box.width)

h = int(box.height)

boxCenterX = x + int(w/2)

boxCenterY = y + int(h/2)

cv2.rectangle(frame,(x,y),(x+w,y+h),

(255,0,0),fontThickness)

labelText = "Target Aquired "+str(round(category.score*100))+'%'

cv2.circle(frame,(boxCenterX,boxCenterY),6,(0,0,255),-1)

cv2.putText(frame,labelText,(x,y-10),fontFace,fontScale*.8,(0,255,0),2)

updateServo(boxCenterX,boxCenterY,W,H)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

myDetector.close()

cv2.destroyAllWindows()

print('Program Terminated')

Technology Tutorials

Tag Archives: Artifiial intellignce

AI on the Edge LESSON 57: Detecting and Tracking Objects with MediaPipe