AI on the Edge LESSON 36: Select Active Camera in OpenCV With Voice Commands

Welcome back, makers, engineers, and AI enthusiasts! In our previous lessons, we built out robust multi-camera setups, streaming feeds from both Raspberry Pi cameras and high-definition USB cameras. But watching four feeds in separate tiles is only half the battle. What happens when you want to interact with your system hands-free?

In this lesson, we are taking our edge vision projects to the next level by integrating voice commands. You will learn how to run a dedicated speech-to-text thread in the background, catch voice triggers safely using a thread-safe queue, and dynamically switch your active main camera feed in OpenCV on the fly—just by speaking.

What You Will Learn in This Lesson

Multi-Threaded Speech Recognition: How to run the stt listener in a separate daemon thread so it never blocks or stutters your high-FPS video processing loop.
Thread-Safe Communication: Using Python’s Queue module to pass voice commands seamlessly from the background listening thread into your main application loop.
Handling STT Variations: Accounting for common speech-to-text homophone variations (like “two”, “to”, and “too”, or “four” and “for”) to make your voice control robust and reliable.
Dynamic Frame Routing: Mapping spoken commands to specific camera streams (piCam1, piCam2, usbCam1, and usbCam2) and updating your primary display window instantly.
Organized Window Layouts: Positioning and sizing multiple OpenCV windows on your desktop workspace for a clean, professional multi-camera dashboard.

Step-by-Step Breakdown of the Script

1. Setting Up the Background Voice Thread

When working with real-time video processing in OpenCV, blocking functions are your worst enemy. If you call a speech recognition listening function directly inside your main while loop, your video frames will freeze while waiting for audio input.

To prevent this, we initialize a background worker function (getCamera) and launch it as a daemon thread:

The thread continuously listens for your voice commands using the Fusion Hat STT module.
Once a command is captured, it strips whitespace, checks for exit triggers (like saying 'quit'), and pushes valid commands directly into our commandQ.

2. Initializing Multiple Cameras

Our setup harnesses the full power of the hardware by combining native Raspberry Pi camera interfaces with standard USB webcams:

Pi Cameras: Configured via Picamera2 at a crisp 1280x720 resolution with RGB888 formatting running smoothly at 60 frames per second.
USB Cameras: Initialized through OpenCV’s VideoCapture class, set to target resolution and optimized for 30 FPS.

3. Processing and Routing Commands in the Main Loop

Inside the primary application loop, our script handles three core tasks simultaneously:

Calculate Performance: Continuously tracks and smooths out the frames-per-second (FPS) metric so you can monitor system load.
Check the Command Queue: Non-blockingly checks if the background thread has dropped a new camera selection into commandQ. If a new command is waiting, it updates the mainCam variable.
Route the Main Frame: Evaluates the active camera string—including clever fallback checks for common voice misinterpretations like 'camera to' or 'camera for'—and assigns the corresponding video stream to mainFrame.

4. Managing Your OpenCV Windows

To give you a complete command-center experience, the script generates a large primary display window for your active camera view, accompanied by a clean row of smaller preview tiles across the bottom of your screen for all four connected feeds.

import cv2
import time
from picamera2 import Picamera2
import threading
from queue import Queue
from fusion_hat.stt import STT
stt = STT('en-us')
commandQ = Queue()
mainCam = 'camera one'

W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)

piCam1 = Picamera2(0)
piCam1.preview_configuration.main.size = RES
piCam1.preview_configuration.main.format = "RGB888"
piCam1.preview_configuration.controls.FrameRate=60
piCam1.preview_configuration.align()
piCam1.configure("preview")
piCam1.start()

piCam2 = Picamera2(1)
piCam2.preview_configuration.main.size = RES
piCam2.preview_configuration.main.format = "RGB888"
piCam2.preview_configuration.controls.FrameRate=60
piCam2.preview_configuration.align()
piCam2.configure("preview")
piCam2.start()

usbCam1 = cv2.VideoCapture(16)
usbCam1.set(cv2.CAP_PROP_FRAME_WIDTH,W)
usbCam1.set(cv2.CAP_PROP_FRAME_HEIGHT,H)
usbCam1.set(cv2.CAP_PROP_FPS, 30)

usbCam2 = cv2.VideoCapture(18)
usbCam2.set(cv2.CAP_PROP_FRAME_WIDTH,W)
usbCam2.set(cv2.CAP_PROP_FRAME_HEIGHT,H)
usbCam2.set(cv2.CAP_PROP_FPS, 30)

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

running = True
def getCamera():
    print('Voice Thread Started')
    global running
    while running:
        myCamera = stt.listen(stream = False)
        myCamera = myCamera.strip()
        print(myCamera)
        if myCamera == 'quit':
            print('Security Camera Offline')
            running = False
            break
        commandQ.put(myCamera)
    print('Thread Terminated')

cameraThread = threading.Thread(target = getCamera,daemon=True)
cameraThread.start()

cv2.namedWindow('Main Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Main Camera',0,65)
cv2.resizeWindow('Main Camera',W,H)

cv2.namedWindow('Camera1',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera1',0,H)
cv2.resizeWindow('Camera1',int(W/4),int(H/4))

cv2.namedWindow('Camera2',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera2',int(W/4),H)
cv2.resizeWindow('Camera2',int(W/4),int(H/4))

cv2.namedWindow('Camera3',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera3',2*int(W/4),H)
cv2.resizeWindow('Camera3',int(W/4),int(H/4))

cv2.namedWindow('Camera4',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera4',3*int(W/4),H)
cv2.resizeWindow('Camera4',int(W/4),int(H/4))

while running:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame1= piCam1.capture_array()
    #frame1=cv2.flip(frame1,-1)
    
    frame2= piCam2.capture_array()
    frame2=cv2.flip(frame2,-1)
    
    _ , frame3 = usbCam1.read()
    _ , frame4 = usbCam2.read()
    mainFrame = frame1
    
    if commandQ.empty() == False:
        mainCam = commandQ.get()
        
    if mainCam == 'camera one':
        mainFrame = frame1
    if mainCam == 'camera two' or mainCam == 'camera to' or mainCam == 'camera too':
        mainFrame = frame2
    if mainCam == 'camera three':
        mainFrame = frame3
    if mainCam == 'camera four' or mainCam == 'camera for':
        mainFrame = frame4
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame1,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Main Camera", mainFrame)
    cv2.imshow("Camera1", frame1)
    cv2.imshow("Camera2", frame2)
    cv2.imshow("Camera3", frame3)
    cv2.imshow("Camera4", frame4)    
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

import cv2

import time

from picamera2 import Picamera2

import threading

from queue import Queue

from fusion_hat.stt import STT

stt = STT('en-us')

commandQ = Queue()

mainCam = 'camera one'

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam1 = Picamera2(0)

piCam1.preview_configuration.main.size = RES

piCam1.preview_configuration.main.format = "RGB888"

piCam1.preview_configuration.controls.FrameRate=60

piCam1.preview_configuration.align()

piCam1.configure("preview")

piCam1.start()

piCam2 = Picamera2(1)

piCam2.preview_configuration.main.size = RES

piCam2.preview_configuration.main.format = "RGB888"

piCam2.preview_configuration.controls.FrameRate=60

piCam2.preview_configuration.align()

piCam2.configure("preview")

piCam2.start()

usbCam1 = cv2.VideoCapture(16)

usbCam1.set(cv2.CAP_PROP_FRAME_WIDTH,W)

usbCam1.set(cv2.CAP_PROP_FRAME_HEIGHT,H)

usbCam1.set(cv2.CAP_PROP_FPS, 30)

usbCam2 = cv2.VideoCapture(18)

usbCam2.set(cv2.CAP_PROP_FRAME_WIDTH,W)

usbCam2.set(cv2.CAP_PROP_FRAME_HEIGHT,H)

usbCam2.set(cv2.CAP_PROP_FPS, 30)

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

running = True

def getCamera():

print('Voice Thread Started')

global running

while running:

myCamera = stt.listen(stream = False)

myCamera = myCamera.strip()

print(myCamera)

if myCamera == 'quit':

print('Security Camera Offline')

running = False

break

commandQ.put(myCamera)

print('Thread Terminated')

cameraThread = threading.Thread(target = getCamera,daemon=True)

cameraThread.start()

cv2.namedWindow('Main Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Main Camera',0,65)

cv2.resizeWindow('Main Camera',W,H)

cv2.namedWindow('Camera1',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera1',0,H)

cv2.resizeWindow('Camera1',int(W/4),int(H/4))

cv2.namedWindow('Camera2',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera2',int(W/4),H)

cv2.resizeWindow('Camera2',int(W/4),int(H/4))

cv2.namedWindow('Camera3',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera3',2*int(W/4),H)

cv2.resizeWindow('Camera3',int(W/4),int(H/4))

cv2.namedWindow('Camera4',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera4',3*int(W/4),H)

cv2.resizeWindow('Camera4',int(W/4),int(H/4))

while running:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame1= piCam1.capture_array()

#frame1=cv2.flip(frame1,-1)

frame2= piCam2.capture_array()

frame2=cv2.flip(frame2,-1)

_ , frame3 = usbCam1.read()

_ , frame4 = usbCam2.read()

mainFrame = frame1

if commandQ.empty() == False:

mainCam = commandQ.get()

if mainCam == 'camera one':

mainFrame = frame1

if mainCam == 'camera two' or mainCam == 'camera to' or mainCam == 'camera too':

mainFrame = frame2

if mainCam == 'camera three':

mainFrame = frame3

if mainCam == 'camera four' or mainCam == 'camera for':

mainFrame = frame4

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame1,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Main Camera", mainFrame)

cv2.imshow("Camera1", frame1)

cv2.imshow("Camera2", frame2)

cv2.imshow("Camera3", frame3)

cv2.imshow("Camera4", frame4)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

Technology Tutorials

AI on the Edge LESSON 36: Select Active Camera in OpenCV With Voice Commands

What You Will Learn in This Lesson

Step-by-Step Breakdown of the Script

1. Setting Up the Background Voice Thread

2. Initializing Multiple Cameras

3. Processing and Routing Commands in the Main Loop

4. Managing Your OpenCV Windows

Making The World a Better Place One High Tech Project at a Time. Enjoy!