Tag Archives: Artificial intelligence

AI on the Edge 38: Using MediaPipe for Face Recognition on the Raspberry Pi 5

August 2, 2026 admin

In this video lesson we introduce you to MediaPipe. The OS we had you flash in LESSON 1 already has the MediaPipe framework installed, and all the needed and working dependencies. If you have installed that OS and not modified it, this and future lessons will work. If you find dependency errors, you might need to reflash the original OS.

MediaPipe is a free, open-source framework developed by Google that makes it much easier to add advanced computer vision and AI features to your Python programs. It is especially popular among developers who use OpenCV because it works seamlessly with it and delivers excellent real-time performance, even on devices like the Raspberry Pi 5.

With MediaPipe, you can quickly add powerful capabilities such as face detection, face mesh (detailed facial landmarks), hand tracking, body pose estimation, and more — all without having to write complex deep learning code from scratch. It comes with pre-trained machine learning models that are optimized for speed, allowing your programs to run smoothly at 30 frames per second or higher.

The biggest advantage for Python + OpenCV users is its simplicity. You capture video frames using OpenCV or picamera2, pass them to MediaPipe for processing, and then draw the results (such as bounding boxes or landmarks) back onto your image using normal OpenCV functions. This combination gives developers a fast and straightforward way to build interactive computer vision projects like face trackers, gesture-controlled robots, or smart camera applications.

In short, MediaPipe acts as a powerful, easy-to-use toolkit that bridges the gap between OpenCV and modern AI vision technology.

We introduce you to MediaPipe using a simple example where we create a faceFinder on our Raspberry Pi 5.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
piCam = Picamera2()
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

faceFinder = mp.solutions.face_detection.FaceDetection(
    model_selection = 0,
    min_detection_confidence = .5)
mpDraw = mp.solutions.drawing_utils

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    #frame=cv2.flip(frame,-1)
    
    rgbFrame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
    results = faceFinder.process(rgbFrame)
    if results.detections:
        for detection in results.detections:
            mpDraw.draw_detection(frame,detection)
    
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

piCam = Picamera2()

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

faceFinder = mp.solutions.face_detection.FaceDetection(

model_selection = 0,

min_detection_confidence = .5)

mpDraw = mp.solutions.drawing_utils

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

#frame=cv2.flip(frame,-1)

rgbFrame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

results = faceFinder.process(rgbFrame)

if results.detections:

for detection in results.detections:

mpDraw.draw_detection(frame,detection)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

NVIDIA, NVIDIA Jetson Orin Nano

Local Voice Control of NVIDIA Jetson Orin Nano with STT: Getting Started with Vosk

July 22, 2026 admin

Engineering Your Own Local Voice Assistant: No Cloud, No Compromise

Most “smart” voice assistants are just glorified remote controls for someone else’s server. Today, we’re changing that. We are going to build a local, offline voice command pipeline. This isn’t just about saving data; it’s about ownership. When you can control your hardware—like opening and closing a farm gate—without an internet connection, you have built a system that is robust, private, and yours to control forever. Today you are going to get Speech to Text up and running on your NVIDIA Jetson Orin Nano running under Jetpack 7.2.

The “Why” Behind the Setup

You might ask, “Why not just use a cloud API?” Because cloud APIs are fragile. They rely on internet stability, external servers, and privacy-invasive data logging. By running Vosk locally, we keep the processing on your hardware (like the NVIDIA Jetson). It’s faster, it works in the middle of a power-isolated homestead, and it’s 100% secure.

Part 1: Preparing the Environment

Before we can make the machine listen, we have to prepare the battlefield. We aren’t just downloading files; we are setting up a stable environment where your dependencies won’t conflict with your OS.

# Create a dedicated directory so our work stays organized
mkdir -p ~/STT
cd ~/STT

# We need the PortAudio headers. Why? Because Python’s 'pyaudio' is just a wrapper.
# Under the hood, it talks to the C-based PortAudio library. If the headers aren't there,
# the library won't compile, and your microphone will never wake up.
sudo apt update
sudo apt install -y portaudio19-dev wget unzip

# Download the lightweight, low-latency English model weights from Vosk
wget https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip

# Unzip the archive and rename the folder cleanly to 'model' so the Python script finds it
unzip vosk-model-small-en-us-0.15.zip
mv vosk-model-small-en-us-0.15 model

# Clean up the downloaded zip file to keep the folder pristine
rm vosk-model-small-en-us-0.15.zip

# Setting up a Virtual Environment (venv) is the hallmark of a pro.
# It traps our project dependencies inside this folder so we don't break our system Python.
python3 -m venv sttVenv
source sttVenv/bin/activate
pip install --upgrade pip

# Vosk does the heavy lifting for speech recognition, and Pyaudio grabs the raw data.
pip install vosk pyaudio

# Create a dedicated directory so our work stays organized

mkdir -p ~/STT

cd ~/STT

# We need the PortAudio headers. Why? Because Python’s 'pyaudio' is just a wrapper.

# Under the hood, it talks to the C-based PortAudio library. If the headers aren't there,

# the library won't compile, and your microphone will never wake up.

sudo apt update

sudo apt install -y portaudio19-dev wget unzip

# Download the lightweight, low-latency English model weights from Vosk

wget https://alphacephei.com/vosk/models/vosk-model-small-en-us-0.15.zip

# Unzip the archive and rename the folder cleanly to 'model' so the Python script finds it

unzip vosk-model-small-en-us-0.15.zip

mv vosk-model-small-en-us-0.15 model

# Clean up the downloaded zip file to keep the folder pristine

rm vosk-model-small-en-us-0.15.zip

# Setting up a Virtual Environment (venv) is the hallmark of a pro.

# It traps our project dependencies inside this folder so we don't break our system Python.

python3 -m venv sttVenv

source sttVenv/bin/activate

pip install --upgrade pip

# Vosk does the heavy lifting for speech recognition, and Pyaudio grabs the raw data.

pip install vosk pyaudio

This is IMPORTANT!

Now you can post the code below. You also have to point Thonny to run in the virtual environment. Open Thonny, and under run – select interpreter. Then you must point it to /home/yourUserName/STT/ttsVenv/bin/python3. For me, my username is pjm, but you put in your user name in path above. Here is what mine looked like:

Part 2: Solving the PipeWire Challenge

The biggest headache in modern Linux audio is PipeWire. If you try to open a microphone stream using a hardcoded sample rate that doesn’t match your hardware, your program won’t just fail—it will segfault. We use the validation script below to programmatically query the hardware, asking it: “What sample rate are you running at?” before we even try to open a stream.

# Import the PyAudio library to handle real-time microphone recording and audio stream manipulation.
import pyaudio
# Import the built-in JSON module to parse the structured text results returned by the speech recognition engine.
import json
# Import the audioop library to perform real-time mathematical operations and downsampling on raw audio byte data.
import audioop
# Import the core Vosk objects: Model loads the AI brain/weights, and KaldiRecognizer handles the acoustic processing.
from vosk import Model, KaldiRecognizer
# Import the standard OS module to reliably build absolute system file paths across different user directories.
import os
state=None

# Define a function to discover the specific hardware index and sample rate of the active system sound server.
def get_pipewire_index():
    # Instantiate a PyAudio session object to query the operating system's hardware configuration list.
    p = pyaudio.PyAudio()
    # Loop through every single available audio device reported by the operating system kernel.
    for i in range(p.get_device_count()):
        # Retrieve a descriptive configuration dictionary for the current audio device index.
        dev = p.get_device_info_by_index(i)
        # Check if the name of the device contains 'pipewire' to target the modern Linux audio subsystem.
        if "pipewire" in dev.get('name', '').lower():
            # Return the valid device index integer and its hardware default sample rate, converted cleanly to an integer.
            return i, int(dev.get('defaultSampleRate'))
    # Return empty values if the loop completes without finding an active PipeWire sound server profile.
    return None, None

# Execute the discovery function to capture both the correct system hardware device index and its native sample rate.
dev_index, native_rate = get_pipewire_index()

# Check if the hardware discovery handshake failed to locate an active PipeWire audio interface.
if dev_index is None:
    # Alert the user on screen that the script cannot communicate with the system's active audio server layer.
    print("Error: Could not find 'pipewire' device.")
    # Stop the script execution immediately since it is impossible to record audio without a valid device hook.
    exit()

# Instantiate a fresh PyAudio session specifically dedicated to opening our incoming recording channel.
p = pyaudio.PyAudio()
# Open the active system recording stream channel using explicit audio hardware parameter configurations.
# PARAMETER MANUAL:
# - format=pyaudio.paInt16: Sets 16-bit Signed Integer PCM format. (LEAVE ALONE - required by the Vosk engine).
# - channels=1: Sets recording to Mono channel audio. (LEAVE ALONE - Vosk processes single-channel input).
# - rate=native_rate: Sets the sample rate to match the mic exactly. (LEAVE ALONE - prevents driver hardware failure).
# - input=True: Configures this stream as an audio capture device. (LEAVE ALONE - required to wake up the microphone).
# - input_device_index=dev_index: Points to our discovered target device. (LEAVE ALONE - ensures we grab the right mic).
# - frames_per_buffer=4096: Controls audio chunk size. (PLAY WITH: Lower values like 2048 reduce latency but risk buffer overflows; higher values like 8192 increase safety but introduce slight latency).
stream = p.open(format=pyaudio.paInt16, channels=1, rate=native_rate, 
                input=True, input_device_index=dev_index, frames_per_buffer=4096)

# Generate an absolute directory path targeting the exact folder containing your extracted acoustic model weights.
# PARAMETER MANUAL:
# - "STT", "model": Represents folder names. (PLAY WITH: Change these only if your physical folder layout uses different names).
model_path = os.path.join(os.path.expanduser("~"), "STT", "model")
# Load the heavy machine learning weights into system memory to initialize the Vosk voice processing brain.
model = Model(model_path)
# Initialize the primary speech recognizer object using the loaded model weights and target processing frequency.
# PARAMETER MANUAL:
# - model: The initialized AI brain variable. (LEAVE ALONE).
# - 16000: Specifies the target processing sample rate in Hertz. (LEAVE ALONE - Vosk models are mathematically trained for exactly 16000Hz).
recognizer = KaldiRecognizer(model, 16000)

# Print a status message to the terminal showing that the handshake succeeded and revealing the mic's operating frequency.
print(f"Listening on PipeWire at {native_rate}Hz...")
# Turn on the hardware recording valve to begin filling up the system input data buffers.
stream.start_stream()

# Set up an exception safety block to intercept manual user exits gracefully without leaving system resources locked up.
try:
    # Launch the infinite real-time audio capture loop to continuously monitor the microphone stream.
    while True:
        # Pull raw binary audio frame bytes directly out of the incoming hardware stream buffer memory block.
        # PARAMETER MANUAL:
        # - 4096: The chunk size to read. (LEAVE ALONE - must perfectly match your stream's frames_per_buffer setting).
        # - exception_on_overflow=False: Prevents Python crashes if the Orin momentarily falls behind. (LEAVE ALONE).
        data = stream.read(4096, exception_on_overflow=False)
        # Downsample the raw audio bytes mathematically to 16000Hz using digital filtering algorithms.
        # PARAMETER MANUAL:
        # - data: Raw input audio byte string. (LEAVE ALONE).
        # - 2: Size of sample width in bytes for 16-bit sound. (LEAVE ALONE).
        # - 1: Input audio channel count. (LEAVE ALONE).
        # - native_rate: The native frequency of your mic. (LEAVE ALONE).
        # - 16000: The required downsampled output frequency. (LEAVE ALONE).
        # - state: The digital filtering conversion state history object. (LEAVE ALONE).
        # CRITICAL NOTE: The 'state' variable must be initialized as 'state = None' before entering this 'try:' block, 
        # otherwise Python will throw a NameError on the very first loop iteration because 'state' does not exist yet.
        data_16k, state = audioop.ratecv(data, 2, 1, native_rate, 16000, state)
        
        # Feed the processed 16kHz audio data block into Vosk to evaluate if a complete spoken phrase has finished.
        if recognizer.AcceptWaveform(data_16k):
            # Parse the structured JSON text string returned by the acoustic model into a standard Python dictionary.
            res = json.loads(recognizer.Result())
            # Check the parsed result dictionary to ensure that the detected text sequence is not completely blank.
            if res.get("text"):
                # Cleanly output the decoded text command straight to the standard system terminal display screen.
                print("Command: "+res['text'])
# Intercept the user pressing Ctrl+C in the terminal to stop the continuous script loop cleanly.
except KeyboardInterrupt:
    # Print a clean closing status notification on a new line to confirm the signal was intercepted.
    print("\nStopped.")
# Execute mandatory clean up instructions that are guaranteed to run no matter how the script terminates or crashes.
finally:
    # Turn off the active recording stream data valve to stop pulling raw information from the audio driver layer.
    stream.stop_stream()
    # Close the recording stream channel completely to drop the active hardware pointer binding handles.
    stream.close()
    # Terminate the parent PyAudio engine instance to release all allocated audio device resources back to the OS.
    p.terminate()

100

101

102

103

104

105

106

107

108

# Import the PyAudio library to handle real-time microphone recording and audio stream manipulation.

import pyaudio

# Import the built-in JSON module to parse the structured text results returned by the speech recognition engine.

import json

# Import the audioop library to perform real-time mathematical operations and downsampling on raw audio byte data.

import audioop

# Import the core Vosk objects: Model loads the AI brain/weights, and KaldiRecognizer handles the acoustic processing.

from vosk import Model, KaldiRecognizer

# Import the standard OS module to reliably build absolute system file paths across different user directories.

import os

state=None

# Define a function to discover the specific hardware index and sample rate of the active system sound server.

def get_pipewire_index():

# Instantiate a PyAudio session object to query the operating system's hardware configuration list.

p = pyaudio.PyAudio()

# Loop through every single available audio device reported by the operating system kernel.

for i in range(p.get_device_count()):

# Retrieve a descriptive configuration dictionary for the current audio device index.

dev = p.get_device_info_by_index(i)

# Check if the name of the device contains 'pipewire' to target the modern Linux audio subsystem.

if "pipewire" in dev.get('name', '').lower():

# Return the valid device index integer and its hardware default sample rate, converted cleanly to an integer.

return i, int(dev.get('defaultSampleRate'))

# Return empty values if the loop completes without finding an active PipeWire sound server profile.

return None, None

# Execute the discovery function to capture both the correct system hardware device index and its native sample rate.

dev_index, native_rate = get_pipewire_index()

# Check if the hardware discovery handshake failed to locate an active PipeWire audio interface.

if dev_index is None:

# Alert the user on screen that the script cannot communicate with the system's active audio server layer.

print("Error: Could not find 'pipewire' device.")

# Stop the script execution immediately since it is impossible to record audio without a valid device hook.

exit()

# Instantiate a fresh PyAudio session specifically dedicated to opening our incoming recording channel.

p = pyaudio.PyAudio()

# Open the active system recording stream channel using explicit audio hardware parameter configurations.

# PARAMETER MANUAL:

# - format=pyaudio.paInt16: Sets 16-bit Signed Integer PCM format. (LEAVE ALONE - required by the Vosk engine).

# - channels=1: Sets recording to Mono channel audio. (LEAVE ALONE - Vosk processes single-channel input).

# - rate=native_rate: Sets the sample rate to match the mic exactly. (LEAVE ALONE - prevents driver hardware failure).

# - input=True: Configures this stream as an audio capture device. (LEAVE ALONE - required to wake up the microphone).

# - input_device_index=dev_index: Points to our discovered target device. (LEAVE ALONE - ensures we grab the right mic).

# - frames_per_buffer=4096: Controls audio chunk size. (PLAY WITH: Lower values like 2048 reduce latency but risk buffer overflows; higher values like 8192 increase safety but introduce slight latency).

stream = p.open(format=pyaudio.paInt16, channels=1, rate=native_rate,

input=True, input_device_index=dev_index, frames_per_buffer=4096)

# Generate an absolute directory path targeting the exact folder containing your extracted acoustic model weights.

# PARAMETER MANUAL:

# - "STT", "model": Represents folder names. (PLAY WITH: Change these only if your physical folder layout uses different names).

model_path = os.path.join(os.path.expanduser("~"), "STT", "model")

# Load the heavy machine learning weights into system memory to initialize the Vosk voice processing brain.

model = Model(model_path)

# Initialize the primary speech recognizer object using the loaded model weights and target processing frequency.

# PARAMETER MANUAL:

# - model: The initialized AI brain variable. (LEAVE ALONE).

# - 16000: Specifies the target processing sample rate in Hertz. (LEAVE ALONE - Vosk models are mathematically trained for exactly 16000Hz).

recognizer = KaldiRecognizer(model, 16000)

# Print a status message to the terminal showing that the handshake succeeded and revealing the mic's operating frequency.

print(f"Listening on PipeWire at {native_rate}Hz...")

# Turn on the hardware recording valve to begin filling up the system input data buffers.

stream.start_stream()

# Set up an exception safety block to intercept manual user exits gracefully without leaving system resources locked up.

try:

# Launch the infinite real-time audio capture loop to continuously monitor the microphone stream.

while True:

# Pull raw binary audio frame bytes directly out of the incoming hardware stream buffer memory block.

# PARAMETER MANUAL:

# - 4096: The chunk size to read. (LEAVE ALONE - must perfectly match your stream's frames_per_buffer setting).

# - exception_on_overflow=False: Prevents Python crashes if the Orin momentarily falls behind. (LEAVE ALONE).

data = stream.read(4096, exception_on_overflow=False)

# Downsample the raw audio bytes mathematically to 16000Hz using digital filtering algorithms.

# PARAMETER MANUAL:

# - data: Raw input audio byte string. (LEAVE ALONE).

# - 2: Size of sample width in bytes for 16-bit sound. (LEAVE ALONE).

# - 1: Input audio channel count. (LEAVE ALONE).

# - native_rate: The native frequency of your mic. (LEAVE ALONE).

# - 16000: The required downsampled output frequency. (LEAVE ALONE).

# - state: The digital filtering conversion state history object. (LEAVE ALONE).

# CRITICAL NOTE: The 'state' variable must be initialized as 'state = None' before entering this 'try:' block,

# otherwise Python will throw a NameError on the very first loop iteration because 'state' does not exist yet.

data_16k, state = audioop.ratecv(data, 2, 1, native_rate, 16000, state)

# Feed the processed 16kHz audio data block into Vosk to evaluate if a complete spoken phrase has finished.

if recognizer.AcceptWaveform(data_16k):

# Parse the structured JSON text string returned by the acoustic model into a standard Python dictionary.

res = json.loads(recognizer.Result())

# Check the parsed result dictionary to ensure that the detected text sequence is not completely blank.

if res.get("text"):

# Cleanly output the decoded text command straight to the standard system terminal display screen.

print("Command: "+res['text'])

# Intercept the user pressing Ctrl+C in the terminal to stop the continuous script loop cleanly.

except KeyboardInterrupt:

# Print a clean closing status notification on a new line to confirm the signal was intercepted.

print("\nStopped.")

# Execute mandatory clean up instructions that are guaranteed to run no matter how the script terminates or crashes.

finally:

# Turn off the active recording stream data valve to stop pulling raw information from the audio driver layer.

stream.stop_stream()

# Close the recording stream channel completely to drop the active hardware pointer binding handles.

stream.close()

# Terminate the parent PyAudio engine instance to release all allocated audio device resources back to the OS.

p.terminate()

Homework: Your Gate Controller

You now have a system that identifies audio input, resamples it to 16kHz, and outputs text. Your assignment: Transform this text output into an action.

I want you to add a conditional statement to the main loop. If the recognized text is “open”, print an ASCII art representation of an open gate. If it’s “close”, print the closed version. This is the first step in closing the loop between your AI and the physical world. Go get ’em, and don’t just copy the code—understand how the data flows from the microphone to your decision logic!

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

July 3, 2026 admin

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

Welcome, Makers!

Well, hello there! It is absolutely fantastic to have you back. I’m Paul McWhorter, and today, we are taking a massive step forward in our AI on the Edge journey.

Up until now, we’ve been hard-coding our color thresholds (those pesky Lower Color and Upper Color values) to tell our camera what to look for. That’s fine for a science experiment, but it’s not exactly “smart,” is it? If the lighting changes, or if we want to track a different colored object, we have to go back into the code and manually edit those numbers.

Not anymore!

In today’s lesson, we are building a tool that lets us teach the AI. We’re going to use the mouse to draw a Region of Interest (ROI) right on our camera feed. The system will look at the pixels inside that box, calculate the average Hue, Saturation, and Value, and automatically set our tracking range for us.

This is the kind of professional-level functionality that turns a hobby project into a true, intelligent machine.

The Concept: From Hard-Coding to Dynamic Learning

The magic happens in our mouseAction function. Instead of just reading pixel values, we are now implementing a “click-and-drag” system:

Click and Hold: We capture the startX and startY coordinates.
Drag: We draw a rectangle in real-time so we can see exactly what area we are selecting.
Release: We take that specific slice of the image, convert it to the HSV color space, and use the cv2.mean() function to find the average color properties.
Auto-Tune: We set our LC (Lower Color) and UC (Upper Color) based on that average.

By doing this, the system learns what “object” we want to track on the fly. It’s elegant, it’s powerful, and it feels like real magic when you see those servos snap onto your target after a quick mouse drag.

What We’ve Accomplished

By the end of this lesson, you will have a system that:

Visually selects an object using the mouse.
Automatically calculates the optimal HSV thresholds for that specific object.
Updates the tracking behavior immediately without needing to stop or re-run the code.
Maintains that professional “Edge” feel, giving you real-time feedback on your FPS and mouse position data.

A Note on the “Edge”

Remember, we aren’t just running code; we are running on hardware. When we calculate the mean of the ROI, we are doing real image processing on the fly. You’ll notice the Composite and Mask windows updated immediately, giving you a visual confirmation that your “teacher” (you!) has successfully guided the “student” (the AI).

This is the power of working with OpenCV and the Raspberry Pi. You are building a system that observes, thinks, and reacts—all in real-time.

Get Ready to Build

Grab your Pi, make sure your servos are ready to go, and let’s get that camera calibrated. You’ve put in the work to get this far, and today is where all that effort starts to feel really rewarding.

I’m incredibly proud of how far you’ve come. Let’s dive in and start building!

Are you ready to see how accurately your Pi can “see” once you’ve given it the ability to learn from your selections?

In the video lesson we developed the following code.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
from fusion_hat.pwm import PWM
from fusion_hat.servo import Servo
panPin = 2
tiltPin = 3
panServo = Servo(panPin)
tiltServo = Servo(tiltPin)
panAngle = 0
tiltAngle = 0
panServo.angle(panAngle)
tiltServo.angle(tiltAngle)

piCam = Picamera2()
W=1280
H=720
tStart = time.time()
fps = 0

xFrameC = int((W-1)/2)
yFrameC = int((H-1)/2)
xBoxC = xFrameC
yBoxC = yFrameC


redPin = 5
greenPin = 6
bluePin = 7
redLED = PWM(redPin)
greenLED = PWM(greenPin)
blueLED = PWM(bluePin)

RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.06))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
xPos = 0
textLowerLeft1 = (int(W*.01),int(H*.06)*2)
textLowerLeft2 = (int(W*.01),int(H*.06)*3)
yPos = 0
valR = 0
valG = 0
valB = 0

Hue = 0
Sat = 0
Val = 0

LC = (25,100,90)
UC = (35,255,255)

frame = None
track = False
drawing = False
makeROI = False
boxColor = (255,255,0)
boxThick = 3
def mouseAction(event, x, y, flags, param):
    global frame, xPos, yPos, Hue, Sat, Val
    global track,startX,startY,endX,endY,makeROI,drawing
    if event == 0:
        xPos = x
        yPos = y
        if frame is not None:
            valB, valG, valR = frame[y,x]
            redLED.pulse_width_percent(int(valR/255*100))
            greenLED.pulse_width_percent(int(valG/255*100/2))
            blueLED.pulse_width_percent(int(valB/255*100/4))
            frameHSV = cv2.cvtColor(frame,cv2.COLOR_BGR2HSV)
            Hue, Sat, Val =frameHSV[y,x]
            if Hue == 60:
                track = True
            if Hue == 0:
                track = False
        if drawing == True:
            endX = x
            endY = y
    if event == 1:
        drawing = True
        makeROI = False
        startX = x
        startY = y
        endX = x
        endY = y 
    if event == 4:
        drawing = False
        makeROI = True
        endX = x
        endY = y                    
cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)

cv2.namedWindow('Mask',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Mask',W,65)
cv2.resizeWindow('Mask',int(W/2),int(H/2))

cv2.namedWindow('Composite',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Composite',W,65+int(H/2)+25)
cv2.resizeWindow('Composite',int(W/2),int(H/2))

cv2.setMouseCallback('Camera',mouseAction)

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    
    if drawing == True:
        cv2.rectangle(frame,(startX,startY),(endX,endY),boxColor,boxThick)
    
    if makeROI == True:
        ROI = frame[startY:endY,startX:endX]
        ROIhsv = cv2.cvtColor(ROI,cv2.COLOR_BGR2HSV)
        AVGhsv = cv2.mean(ROIhsv)
        print(AVGhsv)
        LC = (AVGhsv[0]-15,AVGhsv[1]-50,AVGhsv[2]-50)
        UC = (AVGhsv[0]+15,255,255)
        cv2.imshow("ROI",ROI)
        cv2.moveWindow("ROI",W,65)
        makeROI = False

    
    cv2.circle(frame,(int(.9*W),int(.1*H)),20,(0,255,0),-1)
    cv2.circle(frame,(int(.95*W),int(.1*H)),20,(0,0,255),-1)    
    frameHSV = cv2.cvtColor(frame,cv2.COLOR_BGR2HSV)
    mask=cv2.inRange(frameHSV,LC,UC)
    composite = cv2.bitwise_and(frame, frame, mask=mask)
    
    contours, _ =cv2.findContours(mask,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
    if contours:
        #cv2.drawContours(frame,contours,-1,(255,0,0),3)
        largestContour = max(contours, key = cv2.contourArea)
        area = cv2.contourArea(largestContour)
        if area>100:
            #cv2.drawContours(frame,largestContour,-1,(255,0,0),3)
            x, y, w, h = cv2.boundingRect(largestContour)
            cv2.rectangle(frame, (x,y),(x+w,y+h),(0,255,255),3)
            xBoxC  =  x + int(w/2)
            yBoxC  =  y + int(h/2)
            
            xError = xBoxC - xFrameC
            yError = yBoxC - yFrameC
            
            cv2.circle(frame,(xFrameC,yFrameC),10,(255,0,0),-1)
            cv2.circle(frame,(xBoxC,yBoxC), 30,(0,0,255),3)
            if track == True:
                panAngle = panAngle - xError/50/2
                panServo.angle(int(panAngle))
                time.sleep(.02)
                tiltAngle=tiltAngle + yError/50/2
                tiltServo.angle(int(tiltAngle))
                time.sleep(.02)
                           
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    
    text1 = "Mouse Pos: "+str((xPos,yPos))
    text2 = "Pixel Color: "+str((Hue,Sat,Val))
    cv2.putText(frame,text1,textLowerLeft1,fontFace,fontScale,fontColor,fontThickness)    
    cv2.putText(frame,text2,textLowerLeft2,fontFace,fontScale,fontColor,fontThickness)    
    cv2.imshow("Camera", frame)
    cv2.imshow("Composite",composite)
    cv2.imshow("Mask",mask)

    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
redLED.pulse_width_percent(0)
greenLED.pulse_width_percent(0)
blueLED.pulse_width_percent(0)
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

from fusion_hat.pwm import PWM

from fusion_hat.servo import Servo

panPin = 2

tiltPin = 3

panServo = Servo(panPin)

tiltServo = Servo(tiltPin)

panAngle = 0

tiltAngle = 0

panServo.angle(panAngle)

tiltServo.angle(tiltAngle)

piCam = Picamera2()

W=1280

H=720

tStart = time.time()

fps = 0

xFrameC = int((W-1)/2)

yFrameC = int((H-1)/2)

xBoxC = xFrameC

yBoxC = yFrameC

redPin = 5

greenPin = 6

bluePin = 7

redLED = PWM(redPin)

greenLED = PWM(greenPin)

blueLED = PWM(bluePin)

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.06))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

xPos = 0

textLowerLeft1 = (int(W*.01),int(H*.06)*2)

textLowerLeft2 = (int(W*.01),int(H*.06)*3)

yPos = 0

valR = 0

valG = 0

valB = 0

Hue = 0

Sat = 0

Val = 0

LC = (25,100,90)

UC = (35,255,255)

frame = None

track = False

drawing = False

makeROI = False

boxColor = (255,255,0)

boxThick = 3

def mouseAction(event, x, y, flags, param):

global frame, xPos, yPos, Hue, Sat, Val

global track,startX,startY,endX,endY,makeROI,drawing

if event == 0:

xPos = x

yPos = y

if frame is not None:

valB, valG, valR = frame[y,x]

redLED.pulse_width_percent(int(valR/255*100))

greenLED.pulse_width_percent(int(valG/255*100/2))

blueLED.pulse_width_percent(int(valB/255*100/4))

frameHSV = cv2.cvtColor(frame,cv2.COLOR_BGR2HSV)

Hue, Sat, Val =frameHSV[y,x]

if Hue == 60:

track = True

if Hue == 0:

track = False

if drawing == True:

endX = x

endY = y

if event == 1:

drawing = True

makeROI = False

startX = x

startY = y

endX = x

endY = y

if event == 4:

drawing = False

makeROI = True

endX = x

endY = y

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

cv2.namedWindow('Mask',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Mask',W,65)

cv2.resizeWindow('Mask',int(W/2),int(H/2))

cv2.namedWindow('Composite',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Composite',W,65+int(H/2)+25)

cv2.resizeWindow('Composite',int(W/2),int(H/2))

cv2.setMouseCallback('Camera',mouseAction)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

if drawing == True:

cv2.rectangle(frame,(startX,startY),(endX,endY),boxColor,boxThick)

if makeROI == True:

ROI = frame[startY:endY,startX:endX]

ROIhsv = cv2.cvtColor(ROI,cv2.COLOR_BGR2HSV)

AVGhsv = cv2.mean(ROIhsv)

print(AVGhsv)

LC = (AVGhsv[0]-15,AVGhsv[1]-50,AVGhsv[2]-50)

UC = (AVGhsv[0]+15,255,255)

cv2.imshow("ROI",ROI)

cv2.moveWindow("ROI",W,65)

makeROI = False

cv2.circle(frame,(int(.9*W),int(.1*H)),20,(0,255,0),-1)

cv2.circle(frame,(int(.95*W),int(.1*H)),20,(0,0,255),-1)

frameHSV = cv2.cvtColor(frame,cv2.COLOR_BGR2HSV)

mask=cv2.inRange(frameHSV,LC,UC)

composite = cv2.bitwise_and(frame, frame, mask=mask)

contours, _ =cv2.findContours(mask,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)

if contours:

#cv2.drawContours(frame,contours,-1,(255,0,0),3)

largestContour = max(contours, key = cv2.contourArea)

area = cv2.contourArea(largestContour)

if area>100:

#cv2.drawContours(frame,largestContour,-1,(255,0,0),3)

x, y, w, h = cv2.boundingRect(largestContour)

cv2.rectangle(frame, (x,y),(x+w,y+h),(0,255,255),3)

xBoxC = x + int(w/2)

yBoxC = y + int(h/2)

xError = xBoxC - xFrameC

yError = yBoxC - yFrameC

cv2.circle(frame,(xFrameC,yFrameC),10,(255,0,0),-1)

cv2.circle(frame,(xBoxC,yBoxC), 30,(0,0,255),3)

if track == True:

panAngle = panAngle - xError/50/2

panServo.angle(int(panAngle))

time.sleep(.02)

tiltAngle=tiltAngle + yError/50/2

tiltServo.angle(int(tiltAngle))

time.sleep(.02)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

text1 = "Mouse Pos: "+str((xPos,yPos))

text2 = "Pixel Color: "+str((Hue,Sat,Val))

cv2.putText(frame,text1,textLowerLeft1,fontFace,fontScale,fontColor,fontThickness)

cv2.putText(frame,text2,textLowerLeft2,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.imshow("Composite",composite)

cv2.imshow("Mask",mask)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

redLED.pulse_width_percent(0)

greenLED.pulse_width_percent(0)

blueLED.pulse_width_percent(0)

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 29: Improved Proportional Object Tracking with Pan Tilt Camera

July 1, 2026 admin

AI on the Edge LESSON 29: Improved Proportional Object Tracking with Pan Tilt Camera

Hey everyone, Paul McWhorter here from TopTechBoy.com. Welcome back to our channel, where we learn to build real, intelligent systems on edge hardware. Go ahead and grab yourself a nice hot cup of coffee or a big glass of iced tea, because today we are going to completely revolutionize the way our robotic pan-tilt camera interacts with the physical world.

In Lesson 28, we successfully closed the loop. We got our camera to physically move and track an object using the error signal calculated from our OpenCV bounding box. It worked, but let’s be honest with ourselves: it was clunky. It was a crude, incremental system that moved the camera by exactly one lazy degree at a time, regardless of whether the target was right next to the crosshairs or flying across the room. It was jerky, it hunted back and forth, and it just wasn’t elegant old-school engineering.

Today, we are throwing away that clunky incremental logic and replacing it with something beautiful: Proportional Control.

The Problem with Lazy Incremental Steps

Before we fix our control loop, we need to understand exactly why our previous system struggled. In our last script, we used conditional statements to see if the error was positive or negative, and then adjusted our angles by a fixed step of 1 or -1.

This created two major engineering flaws:

Lagging on Large Errors: If you suddenly jerked the object 400 pixels away from the center, the camera would take forever to catch up because it could only step at a constant speed of one degree per loop iteration.
Hunting and Jitter on Small Errors: When the object finally got close to the center, the camera would overshoot by a full degree, trip the opposite condition, and step back. It would constantly “hunt” back and forth across the target, buzzing your hardware to pieces.

The Elegance of Proportional Control

In real-world automation, we don’t use rigid, conditional step-programming to move hardware. We use mathematics. We want the camera’s reaction to be completely proportional to the size of the mistake it is trying to correct.

If the object is a massive distance away from the center crosshairs, we want the servo to take a massive, aggressive leap to catch up instantly. As the object gets closer and closer to the center, we want the camera to automatically slow down and gently glide into place. When the error drops to zero, the physical adjustment should naturally drop to zero.

The magic of this approach is that it allows us to completely eliminate the bulky conditional statements and artificial deadbands we wrote last time. The algebra naturally handles the direction and magnitude of the movement.

Breaking Down the Math and Logic

To achieve this fluid motion, we take our raw error signal—the distance in pixels between our frame center and the object center—and apply a scaling factor, known in control theory as Gain.

In this updated system design, we take our error and divide it down. Specifically, we divide the pixel error by 50, and then split that in half by dividing by 2. Mathematically, this means we are scaling our pixel error down by a factor of 100.

If your object is 300 pixels off-center, the math calculates an instantaneous adjustment of 3 degrees, quickly snapping the camera toward the target.
If the object is only 10 pixels off-center, the adjustment becomes a tiny fraction of a degree (0.1), smoothly stabilizing the camera track.

Precision Tracking with Floating-Point Variables

Because we are dividing our pixel error down by 100, our angular adjustments will almost always be fractional decimals rather than clean integers. If we tried to store these angles as standard integers, our program would truncate those decimals, completely throwing away our precise micro-adjustments and causing the camera to stall out.

To make this system work perfectly, we maintain our accumulation variables as high-precision floating-point numbers. The script constantly adds and subtracts these fractional updates over time behind the scenes. We only cast the final calculated angle to a clean, rounded integer at the absolute last microsecond right as we pass the position command to the physical servo motors.

Visual Tuning and Smooth Performance

You will notice a massive visual upgrade when running this refined loop. To match our new high-precision math, we tighten up our tracking reticle overlay, shrinking our target circle down from a radius of 40 to a crisp 30 pixels. We also change our dynamic bounding box to a bright, vibrant yellow to make our tracking visually pop on screen.

When you fire up this loop and wave your object around, you will see a night-and-day difference compared to last week. The lazy, robotic stutter is completely gone. The pan-tilt mount tracks with an organic, fluid motion, actively accelerating and decelerating to mirror your movements perfectly.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
from fusion_hat.pwm import PWM
from fusion_hat.servo import Servo
panPin = 2
tiltPin = 3
panServo = Servo(panPin)
tiltServo = Servo(tiltPin)
panAngle = 0
tiltAngle = 0
panServo.angle(panAngle)
tiltServo.angle(tiltAngle)

piCam = Picamera2()
W=1280
H=720
tStart = time.time()
fps = 0

xFrameC = int((W-1)/2)
yFrameC = int((H-1)/2)
xBoxC = xFrameC
yBoxC = yFrameC


redPin = 5
greenPin = 6
bluePin = 7
redLED = PWM(redPin)
greenLED = PWM(greenPin)
blueLED = PWM(bluePin)

RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.06))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
xPos = 0
textLowerLeft1 = (int(W*.01),int(H*.06)*2)
textLowerLeft2 = (int(W*.01),int(H*.06)*3)
yPos = 0
valR = 0
valG = 0
valB = 0

Hue = 0
Sat = 0
Val = 0

LC = (25,100,90)
UC = (35,255,255)

frame = None
def mouseAction(event, x, y, flags, param):
    global frame, xPos, yPos, Hue, Sat, Val
    if event == 0:
        xPos = x
        yPos = y
        if frame is not None:
            valB, valG, valR = frame[y,x]
            redLED.pulse_width_percent(int(valR/255*100))
            greenLED.pulse_width_percent(int(valG/255*100/2))
            blueLED.pulse_width_percent(int(valB/255*100/4))
            frameHSV = cv2.cvtColor(frame,cv2.COLOR_BGR2HSV)
            Hue, Sat, Val =frameHSV[y,x]
cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)

cv2.namedWindow('Mask',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Mask',W,65)
cv2.resizeWindow('Mask',int(W/2),int(H/2))

cv2.namedWindow('Composite',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Composite',W,65+int(H/2)+25)
cv2.resizeWindow('Composite',int(W/2),int(H/2))

cv2.setMouseCallback('Camera',mouseAction)

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    
    frameHSV = cv2.cvtColor(frame,cv2.COLOR_BGR2HSV)
    mask=cv2.inRange(frameHSV,LC,UC)
    composite = cv2.bitwise_and(frame, frame, mask=mask)
    
    contours, _ =cv2.findContours(mask,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
    if contours:
        #cv2.drawContours(frame,contours,-1,(255,0,0),3)
        largestContour = max(contours, key = cv2.contourArea)
        area = cv2.contourArea(largestContour)
        if area>100:
            #cv2.drawContours(frame,largestContour,-1,(255,0,0),3)
            x, y, w, h = cv2.boundingRect(largestContour)
            cv2.rectangle(frame, (x,y),(x+w,y+h),(0,255,255),3)
            xBoxC  =  x + int(w/2)
            yBoxC  =  y + int(h/2)
            
            xError = xBoxC - xFrameC
            yError = yBoxC - yFrameC
            
            cv2.circle(frame,(xFrameC,yFrameC),10,(255,0,0),-1)
            cv2.circle(frame,(xBoxC,yBoxC), 30,(0,0,255),3)
            panAngle = panAngle - xError/50/2
            panServo.angle(int(panAngle))
            time.sleep(.02)
            tiltAngle=tiltAngle + yError/50/2
            tiltServo.angle(int(tiltAngle))
            time.sleep(.02)
                           
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    
    text1 = "Mouse Pos: "+str((xPos,yPos))
    text2 = "Pixel Color: "+str((Hue,Sat,Val))
    cv2.putText(frame,text1,textLowerLeft1,fontFace,fontScale,fontColor,fontThickness)    
    cv2.putText(frame,text2,textLowerLeft2,fontFace,fontScale,fontColor,fontThickness)    
    cv2.imshow("Camera", frame)
    cv2.imshow("Composite",composite)
    cv2.imshow("Mask",mask)

    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
redLED.pulse_width_percent(0)
greenLED.pulse_width_percent(0)
blueLED.pulse_width_percent(0)
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

from fusion_hat.pwm import PWM

from fusion_hat.servo import Servo

panPin = 2

tiltPin = 3

panServo = Servo(panPin)

tiltServo = Servo(tiltPin)

panAngle = 0

tiltAngle = 0

panServo.angle(panAngle)

tiltServo.angle(tiltAngle)

piCam = Picamera2()

W=1280

H=720

tStart = time.time()

fps = 0

xFrameC = int((W-1)/2)

yFrameC = int((H-1)/2)

xBoxC = xFrameC

yBoxC = yFrameC

redPin = 5

greenPin = 6

bluePin = 7

redLED = PWM(redPin)

greenLED = PWM(greenPin)

blueLED = PWM(bluePin)

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.06))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

xPos = 0

textLowerLeft1 = (int(W*.01),int(H*.06)*2)

textLowerLeft2 = (int(W*.01),int(H*.06)*3)

yPos = 0

valR = 0

valG = 0

valB = 0

Hue = 0

Sat = 0

Val = 0

LC = (25,100,90)

UC = (35,255,255)

frame = None

def mouseAction(event, x, y, flags, param):

global frame, xPos, yPos, Hue, Sat, Val

if event == 0:

xPos = x

yPos = y

if frame is not None:

valB, valG, valR = frame[y,x]

redLED.pulse_width_percent(int(valR/255*100))

greenLED.pulse_width_percent(int(valG/255*100/2))

blueLED.pulse_width_percent(int(valB/255*100/4))

frameHSV = cv2.cvtColor(frame,cv2.COLOR_BGR2HSV)

Hue, Sat, Val =frameHSV[y,x]

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

cv2.namedWindow('Mask',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Mask',W,65)

cv2.resizeWindow('Mask',int(W/2),int(H/2))

cv2.namedWindow('Composite',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Composite',W,65+int(H/2)+25)

cv2.resizeWindow('Composite',int(W/2),int(H/2))

cv2.setMouseCallback('Camera',mouseAction)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

frameHSV = cv2.cvtColor(frame,cv2.COLOR_BGR2HSV)

mask=cv2.inRange(frameHSV,LC,UC)

composite = cv2.bitwise_and(frame, frame, mask=mask)

contours, _ =cv2.findContours(mask,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)

if contours:

#cv2.drawContours(frame,contours,-1,(255,0,0),3)

largestContour = max(contours, key = cv2.contourArea)

area = cv2.contourArea(largestContour)

if area>100:

#cv2.drawContours(frame,largestContour,-1,(255,0,0),3)

x, y, w, h = cv2.boundingRect(largestContour)

cv2.rectangle(frame, (x,y),(x+w,y+h),(0,255,255),3)

xBoxC = x + int(w/2)

yBoxC = y + int(h/2)

xError = xBoxC - xFrameC

yError = yBoxC - yFrameC

cv2.circle(frame,(xFrameC,yFrameC),10,(255,0,0),-1)

cv2.circle(frame,(xBoxC,yBoxC), 30,(0,0,255),3)

panAngle = panAngle - xError/50/2

panServo.angle(int(panAngle))

time.sleep(.02)

tiltAngle=tiltAngle + yError/50/2

tiltServo.angle(int(tiltAngle))

time.sleep(.02)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

text1 = "Mouse Pos: "+str((xPos,yPos))

text2 = "Pixel Color: "+str((Hue,Sat,Val))

cv2.putText(frame,text1,textLowerLeft1,fontFace,fontScale,fontColor,fontThickness)

cv2.putText(frame,text2,textLowerLeft2,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.imshow("Composite",composite)

cv2.imshow("Mask",mask)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

redLED.pulse_width_percent(0)

greenLED.pulse_width_percent(0)

blueLED.pulse_width_percent(0)

print('Program Terminated')

NVIDIA

NVIDIA Jetson Orin Nano: Secret to Running Ollama on the GPU

June 18, 2026 admin

One of the biggest frustrations with the new Jetpack 7.2 release is finding out that a standard installation of Ollama—the gold standard for running local LLMs—completely ignores your powerful NVIDIA GPU and defaults to the CPU.

In this lesson, we aren’t just going to fix that; we are going to measure the “truth” behind the performance. We will use data to see exactly how much gain we get from the GPU and where the hardware starts to hit the thermal throttling wall.

The Problem: The “Canned” Installation

When you run a standard Ollama install on the Jetson Orin Nano, the system doesn’t automatically recognize the integrated GPU (iGPU). If you open your NVIDIA Power GUI (jtop), you will see your CPU cores pegged at 100% while the GPU sits idle. This leads to slow response times and a disappointing experience.

Lets start by the standard ‘Canned’ Installation. The good news is, it is very simple:

curl -fsSL https://ollama.com/install.sh | sh

1	curl -fsSL https://ollama.com/install.sh \| sh

To see exactly how your system is performing, run Ollama in verbose mode:

ollama run gemma3:1b

1	ollama run gemma3:1b

At this point you will have Ollama running a simple LLM locally on your Jetson Orin Nano. This is a huge step forward, but we now want to dig deeper and actually see how well this simple model is performing. The first thing we do is run the Jetson Power GUI, hidden behind the NVIDIA icon in upper right of the menu bar.

Pay close attention to the Prompt Eval Rate and Eval Rate (tokens per second). These are our baseline numbers.

The “Secret Sauce” Solution

To force Ollama to use the Jetson’s CUDA cores, we have to manually override the system service configuration.

sudo apt-get update && sudo apt-get install nano

1	sudo apt-get update && sudo apt-get install nano

Step 1: Install the Nano Editor

Before we can edit system files, we need a reliable text editor. If you don’t have it yet, run this command:

Step 2: Create the Service Override

We need to tell the Ollama service exactly where to look for the GPU libraries. Use nano to open the following file:

sudo nano /etc/systemd/system/ollama.service.d/override.conf

1	sudo nano /etc/systemd/system/ollama.service.d/override.conf

Step 3: Add the Configuration

Copy and paste the following block into that file. This is the “Secret Sauce” that enables the iGPU and points the system to the correct CUDA backend:

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_CONTEXT_LENGTH=4096"
Environment="OLLAMA_IGPU_ENABLE=1"
Environment="GGML_BACKEND_PATH=/usr/local/lib/ollama/cuda_v13/libggml-cuda.so"
Environment="LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v13"

[Service]

Environment="OLLAMA_HOST=127.0.0.1:11434"

Environment="OLLAMA_CONTEXT_LENGTH=4096"

Environment="OLLAMA_IGPU_ENABLE=1"

Environment="GGML_BACKEND_PATH=/usr/local/lib/ollama/cuda_v13/libggml-cuda.so"

Environment="LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v13"

Note: Save the file by pressing Ctrl+O, Enter, and then Ctrl+X to exit.

Step 4: Reboot

For the changes to take effect, We will do a reboot.

Benchmarking the Results

Once you have the GPU engaged, the real work begins. In the video, we look at a side-by-side comparison of performance across different Jetson Power Modes (10W, 15W, and MaxN).

Power Level	Prompt Eval Rate (t/s)	Eval Rate (t/s)	Throttling Observed?
CPU	[Your Data]	[Your Data]	Yes/No
10W	[Your Data]	[Your Data]	Yes/No
15W	[Your Data]	[Your Data]	Yes/No
MaxN	[Your Data]	[Your Data]	Yes/No

As we discovered, moving to the GPU provides a boost, but it also increases the heat signature. Watch the full video to see the charts and understand which power level provides the best “sweet spot” for stable, long-term AI performance on your Jetson Orin Nano. This is an important first step . . . getting the heavy lifting down to the GPU. Now in future videos we will explore how to get the work done Well on the GPU.

Technology Tutorials

Tag Archives: Artificial intelligence

AI on the Edge 38: Using MediaPipe for Face Recognition on the Raspberry Pi 5

Local Voice Control of NVIDIA Jetson Orin Nano with STT: Getting Started with Vosk

Engineering Your Own Local Voice Assistant: No Cloud, No Compromise

The “Why” Behind the Setup

Part 1: Preparing the Environment

This is IMPORTANT!

Part 2: Solving the PipeWire Challenge

Homework: Your Gate Controller

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

AI on the Edge LESSON 30: Tune Object Tracker with Mouse Selected ROI

Welcome, Makers!

The Concept: From Hard-Coding to Dynamic Learning

What We’ve Accomplished

A Note on the “Edge”

Get Ready to Build

AI on the Edge LESSON 29: Improved Proportional Object Tracking with Pan Tilt Camera

AI on the Edge LESSON 29: Improved Proportional Object Tracking with Pan Tilt Camera

The Problem with Lazy Incremental Steps

The Elegance of Proportional Control

Breaking Down the Math and Logic

Precision Tracking with Floating-Point Variables

Visual Tuning and Smooth Performance

NVIDIA Jetson Orin Nano: Secret to Running Ollama on the GPU

The Problem: The “Canned” Installation

The “Secret Sauce” Solution

Step 1: Install the Nano Editor

Step 2: Create the Service Override

Step 3: Add the Configuration

Step 4: Reboot

Benchmarking the Results

Making The World a Better Place One High Tech Project at a Time. Enjoy!