AI on the Edge LESSON 58: Object Detection With Speech to Text Voice Control on Raspberry Pi 5

Welcome back, my friends! This is Paul McWhorter from toptechboy.com, and welcome to AI on the Edge LESSON 58: Object Detection With Speech to Text Voice Control on Raspberry Pi 5!

In this lesson, we take a massive leap forward in edge artificial intelligence and robotics by giving our Raspberry Pi 5 camera rig the gift of speech. Up until now, changing our target object of interest required editing our Python script and restarting the code. In Lesson 58, we bridge the gap between computer vision and natural human interaction by introducing local Speech-to-Text (STT) and Piper Text-to-Speech (TTS). Now, you can simply speak to your system, tell it what object you want to track, and watch as your pan-tilt camera locks on and follows your command!

Understanding the Program Architecture

The program is structured around a robust, multi-threaded architecture designed to handle audio recognition and high-speed computer vision simultaneously without lagging your video stream.

1. Voice-Driven Input via Background Threading

To ensure our camera stream maintains a smooth frame rate, we initialize a dedicated background worker thread using Python’s threading library. This thread runs independently, listening to microphone input via local STT tools.

When you speak your object of interest, the thread captures the audio, strips and lowercases the text, and checks for termination commands like quitting the program.
If a valid target is spoken, the thread pushes the string into a thread-safe communication queue and commands Piper TTS to talk back, audibly confirming your choice with a phrase like “Now Tracking”.

2. Thread-Safe Command Queuing

Because audio processing and computer vision operate on different timing loops, we use Python’s thread-safe queue mechanism. The background listening thread drops new target requests into the queue, while our main loop checks on every frame pass whether a new object of interest has arrived. This prevents race conditions and keeps our variable updates clean and reliable.

3. High-Speed OpenCV and MediaPipe Vision Loop

The main execution thread runs our primary while loop, capturing high-resolution frames from the Picamera2 camera module at 60 FPS, flipping the orientation, and feeding the frame data into MediaPipe’s efficient object detection model.

When detections match our active target class, the program extracts bounding box coordinates, calculates the geometric center of the object, renders visual confirmation boxes and confidence percentages on the screen, and computes pixel error offsets from the center of the display.

4. Proportional Servo Feedback Control

Once the pixel error between the target center and the screen center is calculated, our servo update function kicks in. By applying proportional gain multipliers to the horizontal and vertical error values, the script smoothly drives the SunFounder Fusion HAT servos, keeping your target locked right in the crosshairs.

Hardware Requirements

Raspberry Pi 5
Raspberry Pi Camera Module (configured with Picamera2 at 1280 by 720 resolution and 60 FPS)
SunFounder Fusion HAT (for driving pan and tilt servos)
USB Microphone (for local speech-to-text input)
Speaker or Headphones (for Piper text-to-speech audio feedback)

Homework Assignment for Next Week

Now that you have full voice-controlled object selection up and running, it is time for your next big challenge. For next week’s lesson, take the code provided at the bottom of this page and add mode control with three distinct operating states:

find mode: In this mode, the servos remain completely switched off. The camera stays stationary, but MediaPipe actively finds, labels, and boxes your object of interest whenever it appears in the frame.
track mode: In this mode, the servos are switched on, and the camera actively drives the pan-tilt rig to track and center the object of interest.
scan mode: In this mode, if your target is out of view, the system actively sweeps and scans the room using the servos to find the object, and then automatically switches over to track mode once the object is located.

Grab your coffee, fire up your Raspberry Pi 5, review the complete code at the bottom of this page, and I will see you in the next lesson!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
from fusion_hat.servo import Servo

from fusion_hat.stt import STT
from fusion_hat.tts import Piper
import threading
from queue import Queue

tts = Piper()
tts.set_model('en_US-kristin-medium')

stt = STT('en-us')
ooiQueue = Queue()
running = True


piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
topBar = 65
windowPadding = 0
cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)
cv2.resizeWindow('Camera', W, H)
tSettle = .1
time.sleep(tSettle)
cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"
scoreThreshold = .35
maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)
options = vision.ObjectDetectorOptions(base_options = baseOptions,
                                       score_threshold=scoreThreshold,
                                       max_results=maxObj)
myDetector = vision.ObjectDetector.create_from_options(options)

targetClass = None

panPin = 2
tiltPin = 3
panServo = Servo(panPin)
tiltServo = Servo(tiltPin)
panAngle = 0
tiltAngle = 0
panServo.angle(panAngle)
tiltServo.angle(0)

panGain = .015
tiltGain = .015
deadBand = 15

def listenForOoiThread():
    global running
    print("Voice Listener is Running")
    while running:
        spokenText = stt.listen(stream=False)
        if spokenText:
            cleanText = spokenText.strip().lower()
            print("Heard OOI Request: ", cleanText)
            if cleanText == 'quit':
                running = False
                exitMsg = "Terminating Objest Tracking Program, Have a Nice Day"
                tts.say(exitMsg, stream = False)
            ooiQueue.put(cleanText)
            confirmMsg = "Now Tracking: "+cleanText
            tts.say(confirmMsg,stream=False)
    print("Voice Litener Thread Terminated")
                
voiceThread = threading.Thread(target=listenForOoiThread, daemon=False)
voiceThread.start()

def updateServo(objectCenterX,objectCenterY, width, height):
    global panAngle, tiltAngle
    tSettle = .01
    screenCenterX = width/2
    screenCenterY = height/2
    
    errorX = objectCenterX - screenCenterX
    errorY = objectCenterY - screenCenterY
    
    if abs(errorX) > deadBand:
        panAngle = panAngle - (errorX*panGain)
        if panAngle > 90:
            panAngle = 90
        if panAngle < -90:
            panAngle = -90
        panServo.angle(panAngle)
    if abs(errorY) > deadBand:
        tiltAngle = tiltAngle + (errorY*tiltGain)
        if tiltAngle > 45:
            tiltAngle = 45
        if tiltAngle < -45:
            tiltAngle = -45
        tiltServo.angle(tiltAngle)
        time.sleep(tSettle)
tWait = 1
time.sleep(tWait)
welcomeMsg = "Please speak your object of interest to track"  
tts.say(welcomeMsg,stream = False)

while running:
    if not ooiQueue.empty():
        targetClass = ooiQueue.get()
        print("Active OOI Updated to: ",targetClass)
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)
    myDetections = myDetector.detect(myImage)
    targetInZone = False
    #print(myDetections.detections)
    cv2.circle(frame,(int(W/2),int(H/2)),deadBand*2,(0,255,0),2)
    for detection in myDetections.detections:
        if detection.categories:
            category=detection.categories[0]
            if category.category_name == targetClass:
                box = detection.bounding_box
                x = int(box.origin_x)
                y = int(box.origin_y)
                w = int(box.width)
                h = int(box.height)
                boxCenterX = x + int(w/2)
                boxCenterY = y + int(h/2)
                cv2.rectangle(frame,(x,y),(x+w,y+h),
                              (255,0,0),fontThickness)
                labelText = "Target Aquired "+str(round(category.score*100))+'%'
                cv2.circle(frame,(boxCenterX,boxCenterY),6,(0,0,255),-1)
                cv2.putText(frame,labelText,(x,y-10),fontFace,fontScale*.8,(0,255,0),2)
                updateServo(boxCenterX,boxCenterY,W,H)
                                     
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    #cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
myDetector.close()
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

from mediapipe.tasks import python

from mediapipe.tasks.python import vision

from fusion_hat.servo import Servo

from fusion_hat.stt import STT

from fusion_hat.tts import Piper

import threading

from queue import Queue

tts = Piper()

tts.set_model('en_US-kristin-medium')

stt = STT('en-us')

ooiQueue = Queue()

running = True

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

topBar = 65

windowPadding = 0

cv2.namedWindow('Camera', cv2.WINDOW_GUI_NORMAL)

cv2.resizeWindow('Camera', W, H)

tSettle = .1

time.sleep(tSettle)

cv2.moveWindow('Camera', windowPadding, topBar)

modelPath = "/home/pi/ai-lab-kit/mediapipe/efficientdet_lite0.tflite"

scoreThreshold = .35

maxObj = 3

baseOptions = python.BaseOptions(model_asset_path = modelPath)

options = vision.ObjectDetectorOptions(base_options = baseOptions,

score_threshold=scoreThreshold,

max_results=maxObj)

myDetector = vision.ObjectDetector.create_from_options(options)

targetClass = None

panPin = 2

tiltPin = 3

panServo = Servo(panPin)

tiltServo = Servo(tiltPin)

panAngle = 0

tiltAngle = 0

panServo.angle(panAngle)

tiltServo.angle(0)

panGain = .015

tiltGain = .015

deadBand = 15

def listenForOoiThread():

global running

print("Voice Listener is Running")

while running:

spokenText = stt.listen(stream=False)

if spokenText:

cleanText = spokenText.strip().lower()

print("Heard OOI Request: ", cleanText)

if cleanText == 'quit':

running = False

exitMsg = "Terminating Objest Tracking Program, Have a Nice Day"

tts.say(exitMsg, stream = False)

ooiQueue.put(cleanText)

confirmMsg = "Now Tracking: "+cleanText

tts.say(confirmMsg,stream=False)

print("Voice Litener Thread Terminated")

voiceThread = threading.Thread(target=listenForOoiThread, daemon=False)

voiceThread.start()

def updateServo(objectCenterX,objectCenterY, width, height):

global panAngle, tiltAngle

tSettle = .01

screenCenterX = width/2

screenCenterY = height/2

errorX = objectCenterX - screenCenterX

errorY = objectCenterY - screenCenterY

if abs(errorX) > deadBand:

panAngle = panAngle - (errorX*panGain)

if panAngle > 90:

panAngle = 90

if panAngle < -90:

panAngle = -90

panServo.angle(panAngle)

if abs(errorY) > deadBand:

tiltAngle = tiltAngle + (errorY*tiltGain)

if tiltAngle > 45:

tiltAngle = 45

if tiltAngle < -45:

tiltAngle = -45

tiltServo.angle(tiltAngle)

time.sleep(tSettle)

tWait = 1

time.sleep(tWait)

welcomeMsg = "Please speak your object of interest to track"

tts.say(welcomeMsg,stream = False)

while running:

if not ooiQueue.empty():

targetClass = ooiQueue.get()

print("Active OOI Updated to: ",targetClass)

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

myImage = mp.Image(image_format = mp.ImageFormat.SRGB, data=frame)

myDetections = myDetector.detect(myImage)

targetInZone = False

#print(myDetections.detections)

cv2.circle(frame,(int(W/2),int(H/2)),deadBand*2,(0,255,0),2)

for detection in myDetections.detections:

if detection.categories:

category=detection.categories[0]

if category.category_name == targetClass:

box = detection.bounding_box

x = int(box.origin_x)

y = int(box.origin_y)

w = int(box.width)

h = int(box.height)

boxCenterX = x + int(w/2)

boxCenterY = y + int(h/2)

cv2.rectangle(frame,(x,y),(x+w,y+h),

(255,0,0),fontThickness)

labelText = "Target Aquired "+str(round(category.score*100))+'%'

cv2.circle(frame,(boxCenterX,boxCenterY),6,(0,0,255),-1)

cv2.putText(frame,labelText,(x,y-10),fontFace,fontScale*.8,(0,255,0),2)

updateServo(boxCenterX,boxCenterY,W,H)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

#cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

myDetector.close()

cv2.destroyAllWindows()

print('Program Terminated')

Technology Tutorials