Tag Archives: Fusion HAT+

Hey everybody! Welcome back to the AI on the Edge series!

In Lesson 49, we’re doing something that’s just plain cool — we’re turning that tiny little SSD1306 OLED into a live animated avatar that follows your face and hands in real time!

That’s right — no more static images, and no more being tethered to your desktop and monitor. This time your OLED is alive! Move your head, smile, raise your eyebrows, wave your hands… and the little display instantly mirrors your movements like a miniature digital twin. It’s like having a tiny version of yourself living on that 128×64 screen!

What You’ll Learn in This Lesson:

How to use MediaPipe Face Mesh to track 468 facial landmarks
How to combine it with Hand Tracking at the same time
The trick to projecting those landmarks onto a tiny OLED in real time
How to draw both points and connecting lines to create smooth, animated avatars
How to keep everything running fast on the Raspberry Pi

This is one of those projects that makes people stop and say “Wait… how is that even possible on that little screen?!”

By the end of this lesson, you’ll have a foundation for building all kinds of fun real-time avatar projects — everything from simple face mirrors to gesture-controlled characters and beyond.

Why This Is Awesome:

We’re not just displaying video on the big screen anymore — we’re compressing all that AI power down onto a cheap little OLED. This is real AI on the Edge stuff, and it looks incredibly impressive for how simple the hardware is.

Whether you want to build interactive displays, robot faces, wearable tech, or just blow your friends’ minds, this lesson gives you the core technique to make it happen.

Ready to make your OLED come alive?

Fire up the code, run it, and watch your tiny digital self start moving with you. Then start experimenting! Try drawing just the eyes and mouth, or just the hands, or maybe even turn it into a little stick figure avatar.

As always, I want to see what you create with this! Drop your versions in the comments or tag me — I love seeing the creative stuff you guys come up with. Don’t just copy and paste my code. Show you really understand the video by making the code yout own.

Enough talk, lets get this party started!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================

import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
import numpy as np
import board
import adafruit_ssd1306
from PIL import Image, ImageDraw, ImageFont
import time

wOLED = 128
hOLED = 64
I2C_ADDRESS = 0x3C
i2c = board.I2C()
oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

W=1280
H=720
tStart = time.time()
fps = 0

piCam = Picamera2(1)
piCam.preview_configuration.main.size = (W, H)
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate = 60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

faceMesh = mp.solutions.face_mesh.FaceMesh(
    
    max_num_faces=1,
    refine_landmarks=True,
    min_detection_confidence=0.5,
    min_tracking_confidence=0.5)

hands = mp.solutions.hands.Hands(
    model_complexity = 0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands=2 )

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)
while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame = piCam.capture_array()
    frame = cv2.flip(frame, -1)

    rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    faceResults = faceMesh.process(rgb)
    handResults = hands.process(rgb)
    
    image = Image.new("1",(wOLED,hOLED))
    draw = ImageDraw.Draw(image)
    
    if handResults.multi_hand_landmarks:
        for myHands in handResults.multi_hand_landmarks:
            landMarks = myHands.landmark
            lmAll =[]
            for lm in landMarks:
                x = int(lm.x*W)
                y = int(lm.y*H)
                cv2.circle(frame,(x,y),10,(255,0,0),-1)
                xOLED = int(lm.x*wOLED)
                yOLED = int(lm.y*hOLED)
                draw.point((xOLED,yOLED), fill =255)
                lmAll.append((xOLED,yOLED))
            #Uncomment out these lines if you want
            #Lines between the finger joints
#             for idx in mp.solutions.hands.HAND_CONNECTIONS:
#                 lineStart = idx[0]
#                 lineEnd = idx[1]
#                 pt1 = lmAll[lineStart]
#                 pt2 = lmAll[lineEnd]
#                 draw.line((pt1[0],pt1[1],pt2[0],pt2[1]),width=1,fill=255)
#                              
    if faceResults.multi_face_landmarks:
        for faceLandMarks in faceResults.multi_face_landmarks:
            #print(faceLandmarks)
            lmAll = []
            for lm in faceLandMarks.landmark:
                xOLED = int(lm.x*wOLED)
                yOLED = int(lm.y*hOLED)
                draw.point((xOLED,yOLED), fill = 255)
                #print(xOLED,yOLED)
                lmAll.append((xOLED,yOLED))
            for idx in mp.solutions.face_mesh.FACEMESH_CONTOURS:
                #print(idx)
                lineStart = idx[0]
                lineEnd = idx[1]
                pt1 = lmAll[lineStart]
                pt2 = lmAll[lineEnd]
                draw.line((pt1[0],pt1[1],pt2[0],pt2[1]), width=1, fill = 255)



    oled.image(image)
    oled.show()


    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

    cv2.imshow("Camera", frame)

    if cv2.waitKey(1) == ord('q'):
        break

cv2.destroyAllWindows()
piCam.stop()
oled.fill(0)
oled.show()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

import numpy as np

import board

import adafruit_ssd1306

from PIL import Image, ImageDraw, ImageFont

import time

wOLED = 128

hOLED = 64

I2C_ADDRESS = 0x3C

i2c = board.I2C()

oled = adafruit_ssd1306.SSD1306_I2C(wOLED, hOLED, i2c, addr = I2C_ADDRESS)

W=1280

H=720

tStart = time.time()

fps = 0

piCam = Picamera2(1)

piCam.preview_configuration.main.size = (W, H)

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate = 60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

faceMesh = mp.solutions.face_mesh.FaceMesh(

max_num_faces=1,

refine_landmarks=True,

min_detection_confidence=0.5,

min_tracking_confidence=0.5)

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands=2 )

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame = piCam.capture_array()

frame = cv2.flip(frame, -1)

rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

faceResults = faceMesh.process(rgb)

handResults = hands.process(rgb)

image = Image.new("1",(wOLED,hOLED))

draw = ImageDraw.Draw(image)

if handResults.multi_hand_landmarks:

for myHands in handResults.multi_hand_landmarks:

landMarks = myHands.landmark

lmAll =[]

for lm in landMarks:

x = int(lm.x*W)

y = int(lm.y*H)

cv2.circle(frame,(x,y),10,(255,0,0),-1)

xOLED = int(lm.x*wOLED)

yOLED = int(lm.y*hOLED)

draw.point((xOLED,yOLED), fill =255)

lmAll.append((xOLED,yOLED))

#Uncomment out these lines if you want

#Lines between the finger joints

# for idx in mp.solutions.hands.HAND_CONNECTIONS:

# lineStart = idx[0]

# lineEnd = idx[1]

# pt1 = lmAll[lineStart]

# pt2 = lmAll[lineEnd]

# draw.line((pt1[0],pt1[1],pt2[0],pt2[1]),width=1,fill=255)

if faceResults.multi_face_landmarks:

for faceLandMarks in faceResults.multi_face_landmarks:

#print(faceLandmarks)

lmAll = []

for lm in faceLandMarks.landmark:

xOLED = int(lm.x*wOLED)

yOLED = int(lm.y*hOLED)

draw.point((xOLED,yOLED), fill = 255)

#print(xOLED,yOLED)

lmAll.append((xOLED,yOLED))

for idx in mp.solutions.face_mesh.FACEMESH_CONTOURS:

#print(idx)

lineStart = idx[0]

lineEnd = idx[1]

pt1 = lmAll[lineStart]

pt2 = lmAll[lineEnd]

draw.line((pt1[0],pt1[1],pt2[0],pt2[1]), width=1, fill = 255)

oled.image(image)

oled.show()

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

if cv2.waitKey(1) == ord('q'):

break

cv2.destroyAllWindows()

piCam.stop()

oled.fill(0)

oled.show()

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 48: Hand Detection in OpenCV and MediaPipe on the Raspberry Pi

September 6, 2026 admin

In Lesson 48 we’re taking a huge step forward — we’re bringing real-time hand detection to the Raspberry Pi using the incredible power of MediaPipe combined with OpenCV and the Pi Camera. This is one of those projects that feels like pure magic when you see it working: the camera picks up your hands instantly, tracks all 21 landmarks on each hand, draws beautiful connections in real time, and does it all right on the edge with no cloud required.

You’ll learn how to set up MediaPipe’s Hands solution for reliable multi-hand tracking, how to efficiently process frames from the Picamera2 library, flip and convert images for proper display, and draw professional-looking landmarks and connections with custom styling. We also keep a smooth FPS counter running so you can see exactly how well your Pi is performing.

This lesson is exciting because hand tracking is the foundation for so many advanced gesture-control projects — think touchless interfaces, sign language recognition, robotic control, virtual instruments, and interactive art installations. Once you have solid hand detection running smoothly on your Raspberry Pi, the creative possibilities are almost endless.

So grab your Raspberry Pi, fire up the camera, and let’s get those hands dancing on the screen! As always, the full code is waiting for you below. Watch the video, follow along, and then start experimenting — I can’t wait to see what you build with this!

Let’s make something awesome!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================

import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
piCam = Picamera2(1)
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(
    model_complexity =  0,
    min_detection_confidence = .5,
    min_tracking_confidence = .5,
    max_num_hands =2)
mpDraw = mp.solutions.drawing_utils

handStyle = mpDraw.DrawingSpec(
    color = (100, 0, 255),
    thickness = 6,
    circle_radius = 8)
connectionStyle = mpDraw.DrawingSpec(
    color = (0,255,0),
    thickness = 5)

while True:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    results = hands.process(rgbFrame)
    if results.multi_hand_landmarks:
        for handLM in results.multi_hand_landmarks:
            mpDraw.draw_landmarks(
                frame,
                handLM,
                mp.solutions.hands.HAND_CONNECTIONS,
                handStyle,
                connectionStyle)
        
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('Program Terminated')

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

piCam = Picamera2(1)

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

hands = mp.solutions.hands.Hands(

model_complexity = 0,

min_detection_confidence = .5,

min_tracking_confidence = .5,

max_num_hands =2)

mpDraw = mp.solutions.drawing_utils

handStyle = mpDraw.DrawingSpec(

color = (100, 0, 255),

thickness = 6,

circle_radius = 8)

connectionStyle = mpDraw.DrawingSpec(

color = (0,255,0),

thickness = 5)

while True:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

rgbFrame= cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)

results = hands.process(rgbFrame)

if results.multi_hand_landmarks:

for handLM in results.multi_hand_landmarks:

mpDraw.draw_landmarks(

frame,

handLM,

mp.solutions.hands.HAND_CONNECTIONS,

handStyle,

connectionStyle)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('Program Terminated')

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 46: Ultimate Dazzleing Running Rainbow On a NeoPixel Ring

August 30, 2026 admin

In this lesson we create one of the most beautiful and satisfying NeoPixel effects — a smooth, continuous, running rainbow on a 12-LED ring. We affectionately have named this pattern the “Runbow”. This is the “ultimate” version because it is both visually stunning and highly educational, as we explore two fundamentally different programming approaches the Runbow.

The first method is the most intuitive: we place a fixed rainbow across the ring (each LED gets a different hue equally spaced around the color wheel) and then simply rotate the entire pattern one position at a time. This approach is easy to understand because it mimics physically moving a colorful wheel. We save the color of the first pixel, shift all the other pixels one place, and then place the saved color at the end. It feels like a conveyor belt of color circling around the ring.

The second method is more elegant and mathematically pure. Instead of storing and shifting colors, we recalculate the color of every LED on every frame using a moving offset. For each LED we compute its hue as (i / LED_COUNT + offset) % 1.0, where i is the LED’s position and offset is a value that slowly increases over time. This creates a perfectly smooth rainbow that flows around the ring without ever shifting raw RGB values. Because we regenerate the pattern fresh each cycle, there is no risk of color corruption or accumulated errors.

Both techniques produce a gorgeous running rainbow, but they teach very different programming mindsets. The first method helps you deeply understand array manipulation and data movement. The second method introduces the powerful concept of using mathematics and offsets to create motion — a technique used frequently in advanced LED animations, games, and visual effects. In the end, the offset method tends to look smoother and is easier to extend with additional effects (such as brightness pulsing), but both approaches are valuable skills for any embedded AI or IoT developer working with addressable LEDs.You can adjust the speed of the rainbow by changing how much you increment the offset each loop or by modifying the delay. Once you master these two methods, you will be able to create almost any animated pattern you can imagine on your NeoPixel ring. Lets get this party started!

Here is the code we developed in this video:

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import time
import board
import neopixel_spi as neopixel
import colorsys
spi = board.SPI()
LC = 12
PO = neopixel.GRB
strip = neopixel.NeoPixel_SPI(spi,
            LC,pixel_order=PO,
            auto_write=False)
strip.fill(0)
strip.show()
def hsv2rgb(h):
    r, g, b = colorsys.hsv_to_rgb(h,1,1)
    return ( int(255*r), int(255*g),int(255*b))
ts=1
ts2=.1
ts3=.05
try:
    for i in range(12):
        hue = i/12
        color = hsv2rgb(hue)
        strip[i] = color
    strip.show()
    time.sleep(ts)
    for lap in range(70):
        startColor = strip[0]
        for i in range(11):
            strip[i]= strip[i+1]
        strip[11] = startColor
        time.sleep(ts2)
        strip.show()
    offset = 0
    while True:
        for i in range(12):
            hue = (i/12 + offset)%1
            strip[i] = hsv2rgb(hue)
        strip.show()
        offset = (offset + 1/100)%1
        time.sleep(ts3)
    
except KeyboardInterrupt:
    strip.fill(0)
    strip.show()
    print("That's All Folks!")

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import time

import board

import neopixel_spi as neopixel

import colorsys

spi = board.SPI()

LC = 12

PO = neopixel.GRB

strip = neopixel.NeoPixel_SPI(spi,

LC,pixel_order=PO,

auto_write=False)

strip.fill(0)

strip.show()

def hsv2rgb(h):

r, g, b = colorsys.hsv_to_rgb(h,1,1)

return ( int(255*r), int(255*g),int(255*b))

ts=1

ts2=.1

ts3=.05

try:

for i in range(12):

hue = i/12

color = hsv2rgb(hue)

strip[i] = color

strip.show()

time.sleep(ts)

for lap in range(70):

startColor = strip[0]

for i in range(11):

strip[i]= strip[i+1]

strip[11] = startColor

time.sleep(ts2)

strip.show()

offset = 0

while True:

for i in range(12):

hue = (i/12 + offset)%1

strip[i] = hsv2rgb(hue)

strip.show()

offset = (offset + 1/100)%1

time.sleep(ts3)

except KeyboardInterrupt:

strip.fill(0)

strip.show()

print("That's All Folks!")

This is the schematic we are using to connect the NeoPixel ring:

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 45: Adding a NeoPixel Ring To Your Raspberry Pi Project

August 26, 2026 admin

In Lesson 45 of our AI on the Edge series, we take our Fusion AI Lab Kit to the next level by adding a 12-pixel NeoPixel ring. This lesson bridges the gap between pure AI processing and vibrant physical output, showing how your edge AI projects can communicate visually with the real world in a beautiful and engaging way.

We begin by setting up the NeoPixel ring using the SPI interface on the SunFounder Fusion Hat. After importing the necessary libraries (time, board, neopixel_spi, colorsys, and math), we create a simple and reusable hsv2rgb() function that converts Hue values (0.0 to 1.0) into RGB colors that the NeoPixels can understand. This function becomes the foundation for smooth rainbow effects later in the lesson.The lesson starts with basic pixel control. We manually light up each of the 12 pixels one at a time using different colors (red, green, blue, cyan, magenta, yellow, etc.). This slow, deliberate approach lets you clearly see how individual pixel addressing works and helps students understand the coordinate system of the ring.Next, we explore full-ring control by making the entire ring blink between bright red and blue. We then move into motion with a running green pixel moving across a red background — a great introduction to animation techniques. This is followed by a more advanced chasing effect where a blue pixel chases a green pixel around a dim red background.One of the highlights of this lesson is the gentle pulsating aqua effect. Using a sine wave (math.sin), we create a smooth breathing/pulsing animation where the brightness of the color rises and falls naturally. This technique produces a very professional and visually pleasing result that students can easily adapt for future projects.

Finally, we create a rainbow effect. First, we display a uniform rainbow where all 12 pixels show the same color that cycles smoothly through the entire spectrum.

We finish the lesson by assigning the homework. The homework is for you to create the classic running rainbow (what I like to call a “Runbow”), where the colors flow continuously around the ring — one of the most popular and impressive NeoPixel animations.

This lesson reinforces important programming concepts including loops, functions, color theory (HSV vs RGB), timing control, and animation techniques, while giving students an exciting visual payoff. The skills learned here open the door to creating stunning visual feedback for future AI projects — whether it’s status indicators, emotional displays, or attention-grabbing outputs from your edge AI models.

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import time
import board
import neopixel_spi as neopixel
import colorsys
import math

def hsv2rgb(h):
    r, g, b = colorsys.hsv_to_rgb(h, 1, 1)
    return (int(r*255),int(g*255),int(b*255))

spi = board.SPI()
LED_COUNT = 12
PO = neopixel.GRB
strip = neopixel.NeoPixel_SPI(spi,
        LED_COUNT,pixel_order=PO,
        auto_write = False)
strip.fill(0)
strip.show()

strip[0] = (255,0,0)
strip.show()
ts=.1
time.sleep(ts)

strip[1] = (0,255,0)
strip.show()
time.sleep(ts)

strip[2] = (0,0,255)
strip.show()
time.sleep(.1)

strip[3] = (0,255,255)
strip.show()
time.sleep(ts)

strip[4] = (255,0,255)
strip.show()
time.sleep(.1)

strip[5] = (255,255,0)
strip.show()
time.sleep(.1)

strip[6] = (255,100,0)
strip.show()
time.sleep(ts)

strip[7] = (128,0,255)
strip.show()
time.sleep(.1)

strip[8] = (0,255,128)
strip.show()
time.sleep(.1)

strip[9] = (255,0,128)
strip.show()
time.sleep(ts)

strip[10] = (128,255,0)
strip.show()
time.sleep(ts)

strip[11] = (0,128,255)
strip.show()
time.sleep(.1)
ts2=5
for i in range(5):
    strip.fill((255,0,0))
    strip.show()

    time.sleep(ts2)
    
    strip.fill((0,0,255))
    strip.show()
    time.sleep(ts2)
    
strip.fill((255,0,0))
strip.show()
ts3=1
time.sleep(ts3)
ts4=.3
for lap in range(3):
    for i in range(12):
        strip[i] = (0,255,0)
        strip.show()
        time.sleep(ts4)
        strip[i] = (255,0,0)

strip.fill((40,0,0))
strip.show()
time.sleep(ts3)

for lap in range(4):
    for i in range(12):
        greenPos = i
        bluePos = (i + 6)%12
        
        strip[greenPos] = (0,255,0)
        strip[bluePos] = (0,0,255)
        strip.show()
        time.sleep(ts4)
        
        strip[greenPos] = (40,0,0)
        strip[bluePos] = (40,0,0)
        
for laps in range(6):
    for step in range(360):
        brightness= (math.sin(step*3.1415/180)+1)/2
        r = 0
        g = int(255*brightness)
        b = int(50*brightness)
        strip.fill((r,g,b))
        strip.show()
        time.sleep(.005)
        
for laps in range(5):
    for step in range(100):
        hue = step/100
        color = hsv2rgb(hue)
        strip.fill(color)
        strip.show()
        time.sleep(ts)


strip.fill(0)
strip.show()

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import time

import board

import neopixel_spi as neopixel

import colorsys

import math

def hsv2rgb(h):

r, g, b = colorsys.hsv_to_rgb(h, 1, 1)

return (int(r*255),int(g*255),int(b*255))

spi = board.SPI()

LED_COUNT = 12

PO = neopixel.GRB

strip = neopixel.NeoPixel_SPI(spi,

LED_COUNT,pixel_order=PO,

auto_write = False)

strip.fill(0)

strip.show()

strip[0] = (255,0,0)

strip.show()

ts=.1

time.sleep(ts)

strip[1] = (0,255,0)

strip.show()

time.sleep(ts)

strip[2] = (0,0,255)

strip.show()

time.sleep(.1)

strip[3] = (0,255,255)

strip.show()

time.sleep(ts)

strip[4] = (255,0,255)

strip.show()

time.sleep(.1)

strip[5] = (255,255,0)

strip.show()

time.sleep(.1)

strip[6] = (255,100,0)

strip.show()

time.sleep(ts)

strip[7] = (128,0,255)

strip.show()

time.sleep(.1)

strip[8] = (0,255,128)

strip.show()

time.sleep(.1)

strip[9] = (255,0,128)

strip.show()

time.sleep(ts)

strip[10] = (128,255,0)

strip.show()

time.sleep(ts)

strip[11] = (0,128,255)

strip.show()

time.sleep(.1)

ts2=5

for i in range(5):

strip.fill((255,0,0))

strip.show()

time.sleep(ts2)

strip.fill((0,0,255))

strip.show()

time.sleep(ts2)

strip.fill((255,0,0))

strip.show()

ts3=1

time.sleep(ts3)

ts4=.3

for lap in range(3):

for i in range(12):

strip[i] = (0,255,0)

strip.show()

time.sleep(ts4)

strip[i] = (255,0,0)

strip.fill((40,0,0))

strip.show()

time.sleep(ts3)

for lap in range(4):

for i in range(12):

greenPos = i

bluePos = (i + 6)%12

strip[greenPos] = (0,255,0)

strip[bluePos] = (0,0,255)

strip.show()

time.sleep(ts4)

strip[greenPos] = (40,0,0)

strip[bluePos] = (40,0,0)

for laps in range(6):

for step in range(360):

brightness= (math.sin(step*3.1415/180)+1)/2

r = 0

g = int(255*brightness)

b = int(50*brightness)

strip.fill((r,g,b))

strip.show()

time.sleep(.005)

for laps in range(5):

for step in range(100):

hue = step/100

color = hsv2rgb(hue)

strip.fill(color)

strip.show()

time.sleep(ts)

strip.fill(0)

strip.show()

In this class we are using this as our standard components. You should already have the core circuit built and should already have the OLED connected. Today you will add the NeoPixel array.

Our core circuit is:

Fusion Hat Circuit Diagram — This is the circuit we will use moving forward in the class

Last week we also added the SSD1306 OLED Display.

And finally today we add the NeoPixel ring from the Fusion AI Lab Kit.

AI On the Edge, Raspberry Pi

AI on the Edge LESSON 40: Active Face Tracker with Pan Tilt Camera and MediaPipe on Pi 5

August 8, 2026 admin

Boys and girls, welcome back! In today’s lesson, we are going to tie together everything we’ve been building in the AI on the Edge series and construct something truly interactive: a fully autonomous, voice-controlled, pan-tilt face tracking robot running locally right on your Raspberry Pi 5!

In our previous lessons, we learned how to detect faces using MediaPipe and how to drive physical servos to point a camera. Today, we step up our game. We are bringing in multithreading, Speech-to-Text (STT) using the Fusion Hat, and Text-to-Speech (TTS) with Piper to give our Pi a voice, a personality, and the physical ability to track down humanoids in real time.

What We Are Building in This Lesson

Imagine setting up a camera system that constantly scans its environment. The moment a human face enters the frame, the system locks on and speaks up: “Humanoid Detected, Shall I track?”

Using real-time voice commands, you can issue directions straight to the Pi without touching a keyboard:

“Track” — Activates proportional control on the pan-tilt kit. The servos will calculate pixel error relative to the center of the frame and smoothly adjust their angles to keep your face dead center.
“Release” — Disables active tracking, letting the servos hold their position while the vision loop continues monitoring.
“Blind” — Isolates the facial keypoints for the subject’s eyes and draws solid black circles over them in real time, causing the robot to announce: “Subject Has Been Blinded, Shall I Vaporize?”
“Restore” — Removes the eye overlay and brings vision back to normal.
“Quit” — Safely terminates all background threads, announces shutdown, and closes down the application gracefully.

Key Technical Concepts Covered

1. Multi-Threaded Architecture & Thread-Safe Queues

Audio processing—both listening for voice input and generating spoken speech—is computationally heavy and blocking by nature. If you run speech recognition directly inside your primary video processing loop, your frame rate will plummet from a smooth 60 FPS down to a complete crawl.

To solve this, we spin up two independent background threads using Python’s threading module:

Speech Thread: Monitors a thread-safe speakQ (Queue) and handles text-to-speech output using Piper without stalling the main loop.
Command Thread: Continuously listens to the microphone via Speech-to-Text, strips and parses incoming voice triggers, and pushes valid commands into a commandQ.

2. MediaPipe Facial Landmark Detection

We leverage MediaPipe’s high-speed face detection solution running at 1280×720 resolution on the Raspberry Pi 5. By calculating relative bounding boxes and keypoint coordinate matrices (x, y), the system identifies both face centroids and precise feature locations like eye coordinates.

3. Proportional Servo Error Correction

To keep the camera centered on a moving subject, the script computes positional error delta values between the center of the bounding box and the exact midpoint of the camera frame:

xError = xBoxCenter – xFrameCenter

yError = yBoxCenter – yFrameCenter

These error values are scaled down and applied directly to update the current pan and tilt servo angles, ensuring smooth, continuous tracking movement without jarring overshoots.

Your Homework Assignment

Get your Raspberry Pi 5, mount your pan-tilt camera assembly with the Fusion Hat, and implement the multithreaded architecture outlined in this lesson. Tune your servo scaling factors to ensure your tracking motion is fluid and responsive at 60 FPS. Have fun!

# ====================================================================
# DISCLAIMER:
# This code is provided as-is for educational and experimental 
# purposes only. The author makes no representations or warranties of 
# any kind concerning the safety, suitability, or accuracy of this 
# code. Use at your own risk. The author assumes no liability for any 
# damages, system failures, security breaches, or network issues 
# resulting from the use or implementation of this script.
# ====================================================================
 
import cv2
import time
from picamera2 import Picamera2
import mediapipe as mp
#########################################
import threading
from queue import Queue
from fusion_hat.tts import Piper
tts = Piper()
tts.set_model('en_US-kristin-medium')
from fusion_hat.stt import STT
stt = STT('en-us')
 
cnt = 0
running = True
commandQ = Queue()
speakQ = Queue()
###########################################
from fusion_hat.servo import Servo
panPin = 2
tiltPin = 3
panServo = Servo(panPin)
tiltServo = Servo(tiltPin)
panAngle = -10
tiltAngle =-20
panServo.angle(panAngle)
tiltServo.angle(tiltAngle)
piCam = Picamera2()
W=1280
H=720
tStart = time.time()
fps = 0
RES = (W,H)
piCam.preview_configuration.main.size = RES
piCam.preview_configuration.main.format = "RGB888"
piCam.preview_configuration.controls.FrameRate=60
piCam.preview_configuration.align()
piCam.configure("preview")
piCam.start()
textLowerLeft = (int(W*.01),int(H*.05))
fontFace = cv2.FONT_HERSHEY_SIMPLEX
fontThickness = int(W/425)
fontScale = H*.0015
fontColor = (0,0,255)
 
faceFinder = mp.solutions.face_detection.FaceDetection(
    model_selection = 0,
    min_detection_confidence = .5)
 
track = False
xFrameC = int((W-1)/2)
yFrameC = int((H-1)/2)
xBoxC = xFrameC
yBoxC = yFrameC
##########################################
def speak():
    global running
    print('Speak Thread Running')
    while running:
        if speakQ.empty() == False:
            txt = speakQ.get()
            print(txt)
            tts.say(txt)
        time.sleep(.1)
speakThread = threading.Thread(target=speak,daemon=True)
speakThread.start()

def getCommand():
    print('Get Command Thread is Running')
    global running
    while running:
        myCommand = stt.listen(stream=False)
        myCommand = myCommand.strip()
        print(myCommand)
        if myCommand == 'quit':
            msg = 'Project Humanoid Elimination Terminated. Have a Nice Day.'
            speakQ.put(msg)
            running = False
            break
        commandQ.put(myCommand)
    print('Get Command Thread Terminated\n')
commandThread = threading.Thread(target=getCommand,daemon=True)
commandThread.start()
print('Main Program Has Commenced')
msg = 'Hello, I am Ready to Eliminate Humanoids'
speakQ.put(msg)
msgOld = 'Hello, I am Ready to Eliminate Humanoids'
cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)
cv2.moveWindow('Camera',0,65)
cv2.resizeWindow('Camera',W,H)
################################
msg2=''
msg2Old=''
blind = False
while running:
    deltaT = time.time() - tStart
    tStart=time.time()
    fps = fps*.95 + (1/deltaT)*.05
    frame= piCam.capture_array()
    frame=cv2.flip(frame,-1)
    
    if commandQ.empty() == False:
        myCommand = commandQ.get()
        if myCommand == 'track':
            track = True
        if myCommand == 'release':
            track = False
        if myCommand == 'blind':
            blind = True
            msg2="Subject Has Been Blinded, Shall I Vaporize?"
        if myCommand == 'restore':
            blind = False
            msg2 = "Subject Sight Has Been Restored"
        if msg2!= msg2Old:
            speakQ.put(msg2)
        msg2Old = msg2
    rgbFrame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
    results = faceFinder.process(rgbFrame)
    if results.detections:
        msg = "Humanoid Detected, Shall I track"
        for detection in results.detections:
            bbox = detection.location_data.relative_bounding_box
            x = int(bbox.xmin*W)
            y = int(bbox.ymin*H)
            w = int(bbox.width*W)
            h = int(bbox.height*H)
            xBoxC = x + int(w/2)
            yBoxC = y + int(h/2)
            xError = xBoxC - xFrameC
            yError = yBoxC - yFrameC
            cv2.rectangle(frame,
                          (x,y),
                          (x+w,y+h),
                          (0,255,255),
                          3)
            keypoints = detection.location_data.relative_keypoints
            i=0
            for kp in keypoints:
                kpX = int(kp.x*W)
                kpY = int(kp.y*H)
                #cv2.putText(frame,str(i),(kpX,kpY),fontFace,fontScale,fontColor,fontThickness)
                
                if blind == True:
                    if i==0 or i== 1:
                        cv2.circle(frame, (kpX,kpY),
                                   20,
                                   (0,0,0),
                                   -1)
                i=i+1       
    if not results.detections:
        msg = "No Humanoids Detected, Continuing to Scan"
        track = False
    if msgOld != msg:
        speakQ.put(msg)
    msgOld = msg
   
    if track == True and results.detections:
        panAngle = panAngle - xError/50/2
        panServo.angle(int(panAngle))
        #time.sleep(.02)
        tiltAngle = tiltAngle + yError/50/2
        tiltServo.angle(int(tiltAngle))
        #time.sleep(.02) 
    myText = "FPS: "+str(round(fps,1))
    cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)
    cv2.imshow("Camera", frame)
    cv2.moveWindow("Camera",0,60)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()
print('')
print('Program Terminated')

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

# ====================================================================

# DISCLAIMER:

# This code is provided as-is for educational and experimental

# purposes only. The author makes no representations or warranties of

# any kind concerning the safety, suitability, or accuracy of this

# code. Use at your own risk. The author assumes no liability for any

# damages, system failures, security breaches, or network issues

# resulting from the use or implementation of this script.

# ====================================================================

import cv2

import time

from picamera2 import Picamera2

import mediapipe as mp

#########################################

import threading

from queue import Queue

from fusion_hat.tts import Piper

tts = Piper()

tts.set_model('en_US-kristin-medium')

from fusion_hat.stt import STT

stt = STT('en-us')

cnt = 0

running = True

commandQ = Queue()

speakQ = Queue()

###########################################

from fusion_hat.servo import Servo

panPin = 2

tiltPin = 3

panServo = Servo(panPin)

tiltServo = Servo(tiltPin)

panAngle = -10

tiltAngle =-20

panServo.angle(panAngle)

tiltServo.angle(tiltAngle)

piCam = Picamera2()

W=1280

H=720

tStart = time.time()

fps = 0

RES = (W,H)

piCam.preview_configuration.main.size = RES

piCam.preview_configuration.main.format = "RGB888"

piCam.preview_configuration.controls.FrameRate=60

piCam.preview_configuration.align()

piCam.configure("preview")

piCam.start()

textLowerLeft = (int(W*.01),int(H*.05))

fontFace = cv2.FONT_HERSHEY_SIMPLEX

fontThickness = int(W/425)

fontScale = H*.0015

fontColor = (0,0,255)

faceFinder = mp.solutions.face_detection.FaceDetection(

model_selection = 0,

min_detection_confidence = .5)

track = False

xFrameC = int((W-1)/2)

yFrameC = int((H-1)/2)

xBoxC = xFrameC

yBoxC = yFrameC

##########################################

def speak():

global running

print('Speak Thread Running')

while running:

if speakQ.empty() == False:

txt = speakQ.get()

print(txt)

tts.say(txt)

time.sleep(.1)

speakThread = threading.Thread(target=speak,daemon=True)

speakThread.start()

def getCommand():

print('Get Command Thread is Running')

global running

while running:

myCommand = stt.listen(stream=False)

myCommand = myCommand.strip()

print(myCommand)

if myCommand == 'quit':

msg = 'Project Humanoid Elimination Terminated. Have a Nice Day.'

speakQ.put(msg)

running = False

break

commandQ.put(myCommand)

print('Get Command Thread Terminated\n')

commandThread = threading.Thread(target=getCommand,daemon=True)

commandThread.start()

print('Main Program Has Commenced')

msg = 'Hello, I am Ready to Eliminate Humanoids'

speakQ.put(msg)

msgOld = 'Hello, I am Ready to Eliminate Humanoids'

cv2.namedWindow('Camera',cv2.WINDOW_GUI_NORMAL)

cv2.moveWindow('Camera',0,65)

cv2.resizeWindow('Camera',W,H)

################################

msg2=''

msg2Old=''

blind = False

while running:

deltaT = time.time() - tStart

tStart=time.time()

fps = fps*.95 + (1/deltaT)*.05

frame= piCam.capture_array()

frame=cv2.flip(frame,-1)

if commandQ.empty() == False:

myCommand = commandQ.get()

if myCommand == 'track':

track = True

if myCommand == 'release':

track = False

if myCommand == 'blind':

blind = True

msg2="Subject Has Been Blinded, Shall I Vaporize?"

if myCommand == 'restore':

blind = False

msg2 = "Subject Sight Has Been Restored"

if msg2!= msg2Old:

speakQ.put(msg2)

msg2Old = msg2

rgbFrame = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)

results = faceFinder.process(rgbFrame)

if results.detections:

msg = "Humanoid Detected, Shall I track"

for detection in results.detections:

bbox = detection.location_data.relative_bounding_box

x = int(bbox.xmin*W)

y = int(bbox.ymin*H)

w = int(bbox.width*W)

h = int(bbox.height*H)

xBoxC = x + int(w/2)

yBoxC = y + int(h/2)

xError = xBoxC - xFrameC

yError = yBoxC - yFrameC

cv2.rectangle(frame,

(x,y),

(x+w,y+h),

(0,255,255),

keypoints = detection.location_data.relative_keypoints

i=0

for kp in keypoints:

kpX = int(kp.x*W)

kpY = int(kp.y*H)

#cv2.putText(frame,str(i),(kpX,kpY),fontFace,fontScale,fontColor,fontThickness)

if blind == True:

if i==0 or i== 1:

cv2.circle(frame, (kpX,kpY),

20,

(0,0,0),

-1)

i=i+1

if not results.detections:

msg = "No Humanoids Detected, Continuing to Scan"

track = False

if msgOld != msg:

speakQ.put(msg)

msgOld = msg

if track == True and results.detections:

panAngle = panAngle - xError/50/2

panServo.angle(int(panAngle))

#time.sleep(.02)

tiltAngle = tiltAngle + yError/50/2

tiltServo.angle(int(tiltAngle))

#time.sleep(.02)

myText = "FPS: "+str(round(fps,1))

cv2.putText(frame,myText,textLowerLeft,fontFace,fontScale,fontColor,fontThickness)

cv2.imshow("Camera", frame)

cv2.moveWindow("Camera",0,60)

if cv2.waitKey(1)==ord('q'):

break

cv2.destroyAllWindows()

print('')

print('Program Terminated')

Technology Tutorials

Tag Archives: Fusion HAT+

AI on the Edge LESSON 49: Live Animated Avatars on the SSD1306 OLED With OpenCV

What You’ll Learn in This Lesson:

Why This Is Awesome:

AI on the Edge LESSON 48: Hand Detection in OpenCV and MediaPipe on the Raspberry Pi

AI on the Edge LESSON 46: Ultimate Dazzleing Running Rainbow On a NeoPixel Ring

AI on the Edge LESSON 45: Adding a NeoPixel Ring To Your Raspberry Pi Project

AI on the Edge LESSON 40: Active Face Tracker with Pan Tilt Camera and MediaPipe on Pi 5

What We Are Building in This Lesson

Key Technical Concepts Covered

1. Multi-Threaded Architecture & Thread-Safe Queues

2. MediaPipe Facial Landmark Detection

3. Proportional Servo Error Correction

Your Homework Assignment

Making The World a Better Place One High Tech Project at a Time. Enjoy!