Tag Archives: Edge AI

AI on the Edge LESSON 59: Object Detection with Active Scanning in MediaPipe with Pi 5

Hello everyone! Paul McWhorter here from toptechboy.com, and welcome back to our thrilling adventure into computer vision and edge AI. Today, we are dropping Lesson 59 of our AI on the Edge series.

In our previous sessions, we got our Raspberry Pi 5 looking at the world using MediaPipe object detection, we learned how to lock onto targets using pan-and-tilt servos, and we even brought in speech-to-text and text-to-speech capabilities via the Fusion Hat so our system could actually listen and talk back to us.

But there has been one nagging limitation: Static tracking. Up until now, if your object of interest wasn’t sitting right inside the camera’s initial field of view, your Pi just sat there blindly staring straight ahead, waiting for magic to happen.

Well, no more! Today, we are leveling up our system to feature Active Scanning.

The Motivation: From Passive Waiting to Active Hunting

Think about how human vision works. If you are looking for your coffee mug in a messy room, you don’t just stare at the wall and hope it walks past your nose. You move your head, you scan the environment, and your eyes sweep back and forth across different zones until you spot what you are looking for.

That is exactly what we are building in Lesson 59. We are combining three powerful pillars:

  1. Computer Vision (MediaPipe & OpenCV): Running blazing-fast object detection on the Raspberry Pi 5 using the PiCamera2 module.

  2. Robotic Actuation (Pan and Tilt Servos): Giving our camera a physical body with a dynamic range of motion.

  3. Conversational AI (Speech-to-Text & Text-to-Speech): Enabling voice command processing so you can simply tell your Pi what to find, while the system talks back to confirm its state.

How the Active Scanning System Works

In this lesson, our program operates across a few distinct operational modes managed seamlessly through threaded execution:

  • The Voice Listener Thread: Running independently in the background, this thread listens for your spoken commands using the speech-to-text engine. You can say things like “find”, “track”, “scan”, or name a specific object of interest (like “cup”, “person”, or “cell phone”). Saying “quit” gracefully terminates the application with a polite audio farewell.

  • Find Mode: The system rests or holds position while waiting for a target instruction or manual queue update.

  • Scan Mode: If the camera doesn’t immediately see your requested object of interest, it enters active scan mode. The pan servo sweeps smoothly back and forth across its horizontal range, stepping up or down the tilt angle after each pass. It systematically sweeps the room like a radar system.

  • Track Mode: The moment MediaPipe detects the target object with high confidence during a scan, the system instantly halts the scanning pattern, announces “Target Found” via text-to-speech, and switches over to active track mode. Using error correction based on the center of the bounding box versus the center of the screen, the servos continuously adjust to keep the target dead-centered in the frame.

What You Will Learn in the Video

In the accompanying video lesson, we break down the implementation step-by-step:

  • How to manage multi-threaded voice queues so your UI and camera feed don’t stutter while listening for audio.

  • Implementing state-machine logic to toggle cleanly between find, scan, and track modes.

  • Tuning servo gains, deadbands, and sweep step-sizes to achieve smooth, organic robotic movement without jitter.

  • Integrating real-time bounding box feedback, frames-per-second (FPS) monitoring, and dynamic visual overlays using OpenCV.

Grab your Raspberry Pi 5, fire up your Fusion Hat, and let’s write some code that brings hardware to life! Watch the full video lesson above, and let’s dive straight in!

 

AI on the Edge LESSON 37: Using RTSP and IP Cameras in OpenCV on Raspberry Pi 5

The code below shows the work we did in this lesson.

AI on the Edge Lesson 37: Using RTSP and IP Cameras in OpenCV on Raspberry Pi 5

Hey guys! Welcome back to our AI on the Edge series. In our previous lessons, we’ve had a blast working with standard USB webcams, but if you are building a real-world computer vision application, an automation rig, or a security monitoring setup around your home or farm, USB cables just aren’t going to cut it. You need to pull video feeds from remote IP cameras using the Real-Time Streaming Protocol (RTSP).

Today, we are taking that exact step on the Raspberry Pi 5, connecting to an IP camera, streaming the feed smoothly into OpenCV, and—most importantly—solving the dreaded latency problem that plagues RTSP feeds.

The Big Challenge: Conquering RTSP Latency

If you’ve ever tried pulling an RTSP stream into OpenCV straight out of the box, you’ve probably noticed something frustrating: the video lags behind real-time, sometimes by several seconds or even tens of seconds.

Why does that happen? Because by default, FFmpeg and OpenCV buffer incoming frames to ensure smooth playback. But when you are doing computer vision, AI inferencing, or real-time tracking on the edge, you don’t want old history—you want right now.

To fix that, we pass the cv2.CAP_FFMPEG backend flag and immediately flush the buffer by setting the property to 0. This forces OpenCV to drop the backlog and grab the absolute newest frame available from the camera stream, keeping your Pi 5 processing live data in real-time.

Understanding the Script Structure

Let’s break down the key parts of today’s implementation:

  • Credentials & Resolution: We import a separate secret file to keep our camera IP addresses, usernames, and passwords safe and out of public repositories. We lock our resolution at 1280×720 to balance crisp detail with the Pi 5’s processing overhead.
  • Smooth FPS Calculation: Instead of a jittery raw frame-rate readout, we use an exponential moving average to give us a stable, readable performance metric on screen.
  • The Display Window: We configure a GUI window using OpenCV’s window flags so we can easily position and resize our output feed on the desktop.

Drop Your Questions Below

Working with network streams can sometimes be tricky depending on your specific camera’s firmware, codec settings, and network stability. If you run into any connection drops or lag spikes on your Raspberry Pi 5, drop a comment on the video!

Keep building, stay creative, and I will see you guys in Lesson 38!

Here is the code developed in the video

 

Local Voice Control of NVIDIA Jetson Orin Nano with STT: Getting Started with Vosk

Engineering Your Own Local Voice Assistant: No Cloud, No Compromise

Most “smart” voice assistants are just glorified remote controls for someone else’s server. Today, we’re changing that. We are going to build a local, offline voice command pipeline. This isn’t just about saving data; it’s about ownership. When you can control your hardware—like opening and closing a farm gate—without an internet connection, you have built a system that is robust, private, and yours to control forever. Today you are going to get Speech to Text up and running on your NVIDIA Jetson Orin Nano running under Jetpack 7.2.

The “Why” Behind the Setup

You might ask, “Why not just use a cloud API?” Because cloud APIs are fragile. They rely on internet stability, external servers, and privacy-invasive data logging. By running Vosk locally, we keep the processing on your hardware (like the NVIDIA Jetson). It’s faster, it works in the middle of a power-isolated homestead, and it’s 100% secure.

Part 1: Preparing the Environment

Before we can make the machine listen, we have to prepare the battlefield. We aren’t just downloading files; we are setting up a stable environment where your dependencies won’t conflict with your OS.

This is IMPORTANT!

Now you can post the code below. You also have to point Thonny to run in the virtual environment. Open Thonny, and under run –  select interpreter. Then you must point it to /home/yourUserName/STT/ttsVenv/bin/python3. For me, my username is pjm, but you put in your user name in path above. Here is what mine looked like:

Part 2: Solving the PipeWire Challenge

The biggest headache in modern Linux audio is PipeWire. If you try to open a microphone stream using a hardcoded sample rate that doesn’t match your hardware, your program won’t just fail—it will segfault. We use the validation script below to programmatically query the hardware, asking it: “What sample rate are you running at?” before we even try to open a stream.

Homework: Your Gate Controller

You now have a system that identifies audio input, resamples it to 16kHz, and outputs text. Your assignment: Transform this text output into an action.

I want you to add a conditional statement to the main loop. If the recognized text is “open”, print an ASCII art representation of an open gate. If it’s “close”, print the closed version. This is the first step in closing the loop between your AI and the physical world. Go get ’em, and don’t just copy the code—understand how the data flows from the microphone to your decision logic!

AI on the Edge LESSON 33: Tracking Faces with Pan Tilt Camera in OpenCV on Pi 5

Hey guys, Paul McWhorter here from toptechboy.com. In today’s lesson, we are taking our “AI on the Edge” skills to the next level. We aren’t just detecting faces anymore; we are going to make our camera system react to them.

We are integrating our computer vision logic with physical hardware to create a pan-tilt tracking system. We’ll use the Raspberry Pi 5 to run high-speed inference, detect a face, calculate exactly how far that face has drifted from the center of our frame, and then command our servos to follow it in real-time. It is one thing to see a box draw around a face; it is a completely different level of “cool” when the camera actually turns to look at you.

The Engineering Concept: The Error Loop

In robotics, this is a classic control problem. We have a Target (the center of the face) and a Setpoint (the center of our camera frame). The difference between these two points is our Error.

  • xError: How far left or right the face is from the center.
  • yError: How far up or down the face is from the center.

By taking that error and dividing it by a “gain” constant (in our case, 50/2), we can smoothly adjust the servo angles. If we don’t divide by a constant, the camera will snap aggressively to the target or overshoot it. This simple division creates a “proportional” response that keeps our tracking smooth and precise.

What to Focus On

Make sure you have your picamera2 and fusion_hat libraries updated and configured before you dive in. The key to this lesson isn’t just getting the servos to move—it’s understanding how to bridge the gap between the coordinates returned by OpenCV and the angle coordinates required by your servo library.

Pay close attention to how we calculate the center of the frame and the center of the detected face box. Once you understand that math, you can use this same logic to track anything: faces, colored objects, or even specific shapes!

Homework Assignment: Show Your Work!

Alright guys, no excuses! If you want to truly master this hardware, you cannot just sit there and watch me do it—you have to write the code and run it yourself.

Your Task: Get this tracking system working. Once you have it tracking a face, I want you to experiment with that “Gain” factor (the 50/2 part). Try increasing it and decreasing it. What happens to the tracking quality? Is it smoother? Does it jitter?

Record a video of your camera following your face around the room, upload it to YouTube, and link back to this video at the top of your page. Post a link to your homework in the comments section below so I can see you running your code like a boss.

 

The code we developed in the video is available below.

 

NVIDIA Jetson Orin Nano: Secret to Running Ollama on the GPU

One of the biggest frustrations with the new Jetpack 7.2 release is finding out that a standard installation of Ollama—the gold standard for running local LLMs—completely ignores your powerful NVIDIA GPU and defaults to the CPU.

In this lesson, we aren’t just going to fix that; we are going to measure the “truth” behind the performance. We will use data to see exactly how much gain we get from the GPU and where the hardware starts to hit the thermal throttling wall.

The Problem: The “Canned” Installation

When you run a standard Ollama install on the Jetson Orin Nano, the system doesn’t automatically recognize the integrated GPU (iGPU). If you open your NVIDIA Power GUI (jtop), you will see your CPU cores pegged at 100% while the GPU sits idle. This leads to slow response times and a disappointing experience.

Lets start by the standard ‘Canned’ Installation. The good news is, it is very simple:

To see exactly how your system is performing, run Ollama in verbose mode:

At this point you will have Ollama running a simple LLM locally on your Jetson Orin Nano. This is a huge step forward, but we now want to dig deeper and actually see how well this simple model is performing.  The first thing we do is run the Jetson Power GUI, hidden behind the NVIDIA icon in upper right of the menu bar.

Pay close attention to the Prompt Eval Rate and Eval Rate (tokens per second). These are our baseline numbers.

The “Secret Sauce” Solution

To force Ollama to use the Jetson’s CUDA cores, we have to manually override the system service configuration.

Step 1: Install the Nano Editor

Before we can edit system files, we need a reliable text editor. If you don’t have it yet, run this command:

Step 2: Create the Service Override

We need to tell the Ollama service exactly where to look for the GPU libraries. Use nano to open the following file:

Step 3: Add the Configuration

Copy and paste the following block into that file. This is the “Secret Sauce” that enables the iGPU and points the system to the correct CUDA backend:

Note: Save the file by pressing Ctrl+O, Enter, and then Ctrl+X to exit.

Step 4: Reboot

For the changes to take effect, We will do a reboot.

Benchmarking the Results

Once you have the GPU engaged, the real work begins. In the video, we look at a side-by-side comparison of performance across different Jetson Power Modes (10W, 15W, and MaxN).

Power Level Prompt Eval Rate (t/s)  Eval Rate (t/s) Throttling Observed?
CPU [Your Data] [Your Data] Yes/No
10W [Your Data] [Your Data] Yes/No
15W [Your Data] [Your Data] Yes/No
MaxN [Your Data] [Your Data] Yes/No

As we discovered, moving to the GPU provides a boost, but it also increases the heat signature. Watch the full video to see the charts and understand which power level provides the best “sweet spot” for stable, long-term AI performance on your Jetson Orin Nano. This is an important first step . . . getting the heavy lifting down to the GPU. Now in future videos we will explore how to get the work done Well on the GPU.