AI on the Edge LESSON 40: Active Face Tracker with Pan Tilt Camera and MediaPipe on Pi 5

Boys and girls, welcome back! In today’s lesson, we are going to tie together everything we’ve been building in the AI on the Edge series and construct something truly interactive: a fully autonomous, voice-controlled, pan-tilt face tracking robot running locally right on your Raspberry Pi 5!

In our previous lessons, we learned how to detect faces using MediaPipe and how to drive physical servos to point a camera. Today, we step up our game. We are bringing in multithreading, Speech-to-Text (STT) using the Fusion Hat, and Text-to-Speech (TTS) with Piper to give our Pi a voice, a personality, and the physical ability to track down humanoids in real time.

What We Are Building in This Lesson

Imagine setting up a camera system that constantly scans its environment. The moment a human face enters the frame, the system locks on and speaks up: “Humanoid Detected, Shall I track?”

Using real-time voice commands, you can issue directions straight to the Pi without touching a keyboard:

“Track” — Activates proportional control on the pan-tilt kit. The servos will calculate pixel error relative to the center of the frame and smoothly adjust their angles to keep your face dead center.
“Release” — Disables active tracking, letting the servos hold their position while the vision loop continues monitoring.
“Blind” — Isolates the facial keypoints for the subject’s eyes and draws solid black circles over them in real time, causing the robot to announce: “Subject Has Been Blinded, Shall I Vaporize?”
“Restore” — Removes the eye overlay and brings vision back to normal.
“Quit” — Safely terminates all background threads, announces shutdown, and closes down the application gracefully.

Key Technical Concepts Covered

1. Multi-Threaded Architecture & Thread-Safe Queues

Audio processing—both listening for voice input and generating spoken speech—is computationally heavy and blocking by nature. If you run speech recognition directly inside your primary video processing loop, your frame rate will plummet from a smooth 60 FPS down to a complete crawl.

To solve this, we spin up two independent background threads using Python’s threading module:

Speech Thread: Monitors a thread-safe speakQ (Queue) and handles text-to-speech output using Piper without stalling the main loop.
Command Thread: Continuously listens to the microphone via Speech-to-Text, strips and parses incoming voice triggers, and pushes valid commands into a commandQ.

2. MediaPipe Facial Landmark Detection

We leverage MediaPipe’s high-speed face detection solution running at 1280×720 resolution on the Raspberry Pi 5. By calculating relative bounding boxes and keypoint coordinate matrices (x, y), the system identifies both face centroids and precise feature locations like eye coordinates.

3. Proportional Servo Error Correction

To keep the camera centered on a moving subject, the script computes positional error delta values between the center of the bounding box and the exact midpoint of the camera frame:

xError = xBoxCenter – xFrameCenter

yError = yBoxCenter – yFrameCenter

These error values are scaled down and applied directly to update the current pan and tilt servo angles, ensuring smooth, continuous tracking movement without jarring overshoots.

Your Homework Assignment

Get your Raspberry Pi 5, mount your pan-tilt camera assembly with the Fusion Hat, and implement the multithreaded architecture outlined in this lesson. Tune your servo scaling factors to ensure your tracking motion is fluid and responsive at 60 FPS!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
#########################################
import threading
from queue import Queue
from fusion_hat.tts import Piper
tts = Piper()
tts.set_model('en_US-kristin-medium')
from fusion_hat.stt import STT
stt = STT('en-us')
 
cnt = 0
running = True
commandQ = Queue()
speakQ = Queue()
###########################################
from fusion_hat.servo import Servo
panPin = 2
tiltPin = 3
panServo = Servo(panPin)
tiltServo = Servo(tiltPin)
panAngle = -10
tiltAngle =-20
panServo.angle(panAngle)
tiltServo.angle(tiltAngle)
piCam = Picamera2()
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()
textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
 
faceFinder = mp.solutions.face_detection.FaceDetection(
    model_selection = 0,
    min_detection_confidence = .5)
 
track = False
xFrameC = int((W-1)/2)
yFrameC = int((H-1)/2)
xBoxC = xFrameC
yBoxC = yFrameC
##########################################
def speak():
    global running
    print('Speak Thread Running')
    while running:
        if speakQ.empty() == False:
            txt = speakQ.get()
            print(txt)
            tts.say(txt)

        time.sleep(.1)
speakThread = threading.Thread(target=speak,daemon=True)
speakThread.start()
commandThread = threading.Thread(target=getCommand,daemon=True)
commandThread.start()
print('Main Program Has Commenced')
msg = 'Hello, I am Ready to Eliminate Humanoids'
speakQ.put(msg)
msgOld = 'Hello, I am Ready to Eliminate Humanoids'
cv2.namedWindow(' Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)
################################
msg2=''
msg2Old=''
blind = False
while running:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    
    if commandQ.empty() == False:
        myCommand = commandQ.get()
        if myCommand == 'track':
            track = True
        if myCommand == 'release':
            track = False
        if myCommand == 'blind':
            blind = True
            msg2="Subject Has Been Blinded, Shall I Vaporize?"
        if myCommand == 'restore':
            blind = False
            msg2 = "Subject Sight Has Been Restored"
        if msg2!= msg2Old:
            speakQ.put(msg2)
        msg2Old = msg2
    rgbFrame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
    results = faceFinder.process(rgbFrame)
    if results.detections:
        msg = "Humanoid Detected, Shall I track"
        for detection in results.detections:
            bbox = detection.location_data.relative_bounding_box
            x = int(bbox.xmin*W)
            y = int(bbox.ymin*H)
            w = int(bbox.width*W)
            h = int(bbox.height*H)
            xBoxC = x + int(w/2)
            yBoxC = y + int(h/2)
            xError = xBoxC - xFrameC
            yError = yBoxC - yFrameC
            cv2.rectangle(frame,
                          (x,y),
                          (x+w,y+h),
                          (0,255,255),
                          3)
            keypoints = detection.location_data.relative_keypoints
            i=0
            for kp in keypoints:
                kpX = int(kp.x*W)
                kpY = int(kp.y*H)
                #cv2.putText(frame,str(i),(kpX,kpY),fontFace,fontScale,fontColor,fontThickness)
                
                if blind == True:
                    if i==0 or i== 1:
                        cv2.circle(frame, (kpX,kpY),
                                   20,
                                   (0,0,0),
                                   -1)
                i=i+1       
    if not results.detections:
        msg = "No Humanoids Detected, Continuing to Scan"
        track = False
    if msgOld != msg:
        speakQ.put(msg)
    msgOld = msg
   
    if track == True and results.detections:
        panAngle = panAngle - xError/50/2
        panServo.angle(int(panAngle))
        #time.sleep(.02)
        tiltAngle = tiltAngle + yError/50/2
        tiltServo.angle(int(tiltAngle))
        #time.sleep(.02) 
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('')
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

#########################################

import threading

from queue import Queue

from fusion_hat.tts import Piper

tts = Piper()

tts.set_model('en_US-kristin-medium')

from fusion_hat.stt import STT

stt = STT('en-us')

cnt = 0

running = True

commandQ = Queue()

speakQ = Queue()

###########################################

from fusion_hat.servo import Servo

panPin = 2

tiltPin = 3

panServo = Servo(panPin)

tiltServo = Servo(tiltPin)

panAngle = -10

tiltAngle =-20

panServo.angle(panAngle)

tiltServo.angle(tiltAngle)

piCam = Picamera2()

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

faceFinder = mp.solutions.face_detection.FaceDetection(

model_selection = 0,

min_detection_confidence = .5)

track = False

xFrameC = int((W-1)/2)

yFrameC = int((H-1)/2)

xBoxC = xFrameC

yBoxC = yFrameC

##########################################

def speak():

global running

print('Speak Thread Running')

while running:

if speakQ.empty() == False:

txt = speakQ.get()

print(txt)

tts.say(txt)

time.sleep(.1)

speakThread = threading.Thread(target=speak,daemon=True)

speakThread.start()

commandThread = threading.Thread(target=getCommand,daemon=True)

commandThread.start()

print('Main Program Has Commenced')

msg = 'Hello, I am Ready to Eliminate Humanoids'

speakQ.put(msg)

msgOld = 'Hello, I am Ready to Eliminate Humanoids'

cv2.namedWindow(' Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

################################

msg2=''

msg2Old=''

blind = False

while running:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

if commandQ.empty() == False:

myCommand = commandQ.get()

if myCommand == 'track':

track = True

if myCommand == 'release':

track = False

if myCommand == 'blind':

blind = True

msg2="Subject Has Been Blinded, Shall I Vaporize?"

if myCommand == 'restore':

blind = False

msg2 = "Subject Sight Has Been Restored"

if msg2!= msg2Old:

speakQ.put(msg2)

msg2Old = msg2

rgbFrame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

results = faceFinder.process(rgbFrame)

if results.detections:

msg = "Humanoid Detected, Shall I track"

for detection in results.detections:

bbox = detection.location_data.relative_bounding_box

x = int(bbox.xmin*W)

y = int(bbox.ymin*H)

w = int(bbox.width*W)

h = int(bbox.height*H)

xBoxC = x + int(w/2)

yBoxC = y + int(h/2)

xError = xBoxC - xFrameC

yError = yBoxC - yFrameC

cv2.rectangle(frame,

(x,y),

(x+w,y+h),

(0,255,255),

keypoints = detection.location_data.relative_keypoints

i=0

for kp in keypoints:

kpX = int(kp.x*W)

kpY = int(kp.y*H)

#cv2.putText(frame,str(i),(kpX,kpY),fontFace,fontScale,fontColor,fontThickness)

if blind == True:

if i==0 or i== 1:

cv2.circle(frame, (kpX,kpY),

20,

(0,0,0),

-1)

i=i+1

if not results.detections:

msg = "No Humanoids Detected, Continuing to Scan"

track = False

if msgOld != msg:

speakQ.put(msg)

msgOld = msg

if track == True and results.detections:

panAngle = panAngle - xError/50/2

panServo.angle(int(panAngle))

#time.sleep(.02)

tiltAngle = tiltAngle + yError/50/2

tiltServo.angle(int(tiltAngle))

#time.sleep(.02)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('')

print('Program Terminated')

Technology Tutorials

AI on the Edge LESSON 40: Active Face Tracker with Pan Tilt Camera and MediaPipe on Pi 5

What We Are Building in This Lesson

Key Technical Concepts Covered

1. Multi-Threaded Architecture & Thread-Safe Queues

2. MediaPipe Facial Landmark Detection

3. Proportional Servo Error Correction

Your Homework Assignment

Making The World a Better Place One High Tech Project at a Time. Enjoy!