Tag Archives: NVIDIA

NVIDIA Jetson Orin Nano: Secret to Running Ollama on the GPU

June 18, 2026 admin

One of the biggest frustrations with the new Jetpack 7.2 release is finding out that a standard installation of Ollama—the gold standard for running local LLMs—completely ignores your powerful NVIDIA GPU and defaults to the CPU.

In this lesson, we aren’t just going to fix that; we are going to measure the “truth” behind the performance. We will use data to see exactly how much gain we get from the GPU and where the hardware starts to hit the thermal throttling wall.

The Problem: The “Canned” Installation

When you run a standard Ollama install on the Jetson Orin Nano, the system doesn’t automatically recognize the integrated GPU (iGPU). If you open your NVIDIA Power GUI (jtop), you will see your CPU cores pegged at 100% while the GPU sits idle. This leads to slow response times and a disappointing experience.

Lets start by the standard ‘Canned’ Installation. The good news is, it is very simple:

curl -fsSL https://ollama.com/install.sh | sh

1	curl -fsSL https://ollama.com/install.sh \| sh

To see exactly how your system is performing, run Ollama in verbose mode:

ollama run gemma3:1b

1	ollama run gemma3:1b

At this point you will have Ollama running a simple LLM locally on your Jetson Orin Nano. This is a huge step forward, but we now want to dig deeper and actually see how well this simple model is performing. The first thing we do is run the Jetson Power GUI, hidden behind the NVIDIA icon in upper right of the menu bar.

Pay close attention to the Prompt Eval Rate and Eval Rate (tokens per second). These are our baseline numbers.

The “Secret Sauce” Solution

To force Ollama to use the Jetson’s CUDA cores, we have to manually override the system service configuration.

sudo apt-get update && sudo apt-get install nano

1	sudo apt-get update && sudo apt-get install nano

Step 1: Install the Nano Editor

Before we can edit system files, we need a reliable text editor. If you don’t have it yet, run this command:

Step 2: Create the Service Override

We need to tell the Ollama service exactly where to look for the GPU libraries. Use nano to open the following file:

sudo nano /etc/systemd/system/ollama.service.d/override.conf

1	sudo nano /etc/systemd/system/ollama.service.d/override.conf

Step 3: Add the Configuration

Copy and paste the following block into that file. This is the “Secret Sauce” that enables the iGPU and points the system to the correct CUDA backend:

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_CONTEXT_LENGTH=4096"
Environment="OLLAMA_IGPU_ENABLE=1"
Environment="GGML_BACKEND_PATH=/usr/local/lib/ollama/cuda_v13/libggml-cuda.so"
Environment="LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v13"

[Service]

Environment="OLLAMA_HOST=127.0.0.1:11434"

Environment="OLLAMA_CONTEXT_LENGTH=4096"

Environment="OLLAMA_IGPU_ENABLE=1"

Environment="GGML_BACKEND_PATH=/usr/local/lib/ollama/cuda_v13/libggml-cuda.so"

Environment="LD_LIBRARY_PATH=/usr/local/lib/ollama:/usr/local/lib/ollama/cuda_v13"

Note: Save the file by pressing Ctrl+O, Enter, and then Ctrl+X to exit.

Step 4: Reboot

For the changes to take effect, We will do a reboot.

Benchmarking the Results

Once you have the GPU engaged, the real work begins. In the video, we look at a side-by-side comparison of performance across different Jetson Power Modes (10W, 15W, and MaxN).

Power Level	Prompt Eval Rate (t/s)	Eval Rate (t/s)	Throttling Observed?
CPU	[Your Data]	[Your Data]	Yes/No
10W	[Your Data]	[Your Data]	Yes/No
15W	[Your Data]	[Your Data]	Yes/No
MaxN	[Your Data]	[Your Data]	Yes/No

As we discovered, moving to the GPU provides a boost, but it also increases the heat signature. Watch the full video to see the charts and understand which power level provides the best “sweet spot” for stable, long-term AI performance on your Jetson Orin Nano. This is an important first step . . . getting the heavy lifting down to the GPU. Now in future videos we will explore how to get the work done Well on the GPU.

Jetson Nano

AI on the Jetson Nano LESSON 62: Create a Streaming IP Camera from a Raspberry Pi Zero W

October 3, 2020 admin

In this lesson we learn to make a streaming IP camera with a Raspberry Pi Zero W, and Raspberry Pi camera. The Pi will create a RTP stream, which can then be read by a Jetson Nano on the same network. We use OpenCV to read the frames on the NVIDIA Jetson Nano side.

This is the command to launch the Raspberry Pi camera, and start the RTP stream. This command works well for the Raspberry Pi Camera, version 1.

raspivid -t 0 -w 1296 -h 730 -fps 30 -b 2000000 -awb auto -n  -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=0.0.0.0 port=8554

1	raspivid -t 0 -w 1296 -h 730 -fps 30 -b 2000000 -awb auto -n -o - \| gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=0.0.0.0 port=8554

For the Version 2 Camera, I recommend:

raspivid -t 0 -w 1280 -h 720 -fps 30 -b 2000000 -awb auto -n -o - | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=0.0.0.0 port=8554

1	raspivid -t 0 -w 1280 -h 720 -fps 30 -b 2000000 -awb auto -n -o - \| gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=0.0.0.0 port=8554

This is the Gstreamer code on the Jetson Nano side to grab the RTP Frames. On host= below, be sure to use the IP address of your Raspberry Pi.

camSet2=' tcpclientsrc host=10.1.37.28 port=8554 ! gdpdepay ! rtph264depay ! h264parse ! nvv4l2decoder  ! nvvidconv flip-method='+str(flip)+' ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw, width='+str(dispW)+', height='+str(dispH)+',format=BGR ! appsink  drop=true sync=false '

1	camSet2=' tcpclientsrc host=10.1.37.28 port=8554 ! gdpdepay ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv flip-method='+str(flip)+' ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw, width='+str(dispW)+', height='+str(dispH)+',format=BGR ! appsink drop=true sync=false '

Jetson Nano

AI on the Jetson Nano LESSON 61: Image Recognition and Speech (TTS) on the Nano

September 26, 2020 admin

In this video lesson we learn how to add speech to our NVIDIA Jetson Nano we demonstrate how the Jetson can not only recognize an item, but can audibly speak the item it sees. The video takes you through the process step-by-step, and shows you how to make it all work together properly. For your convenience, the code we developed is included below.

import jetson.inference
import jetson.utils
import numpy as np 
import time
import os
from gtts import gTTS
import threading

speak=True
item='Welcome to My Identify. Are you Ready to Rumble?'
confidence=0
itemOld=''


import cv2
print(cv2.__version__)
width=1280
height=720
flip=2
#Uncomment These next Two Line for Pi Camera
#camSet='nvarguscamerasrc !  video/x-raw(memory:NVMM), width=3264, height=2464, format=NV12, framerate=21/1 ! nvvidconv flip-method='+str(flip)+' ! video/x-raw, width='+str(dispW)+', height='+str(dispH)+', format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink'
#cam= cv2.VideoCapture(camSet)

def sayItem():
    global speak
    global item
    while True:
        if speak ==True:
            output=gTTS(text=item, lang='en',slow=False)
            output.save('output.mp3')
            os.system('mpg123 output.mp3')
            speak=False
x=threading.Thread(target=sayItem, daemon=True)
x.start()

#Or, if you have a WEB cam, uncomment the next line
#(If it does not work, try setting to '1' instead of '0')
cam=cv2.VideoCapture('/dev/video1')
cam.set(cv2.CAP_PROP_FRAME_WIDTH,width)
cam.set(cv2.CAP_PROP_FRAME_HEIGHT,height)
net=jetson.inference.imageNet('googlenet')
font=cv2.FONT_HERSHEY_SIMPLEX
timeMark=time.time()
fpsFilter=0

while True:
    ret, frame = cam.read()
    img=cv2.cvtColor(frame,cv2.COLOR_BGR2RGBA).astype(np.float32)
    img=jetson.utils.cudaFromNumpy(img)
    if speak==False:
        classID, confidence = net.Classify(img,width,height)
        if confidence>=.5:
            item=net.GetClassDesc(classID)
            if item!=itemOld:
                speak=True
        if confidence<.5:
            item=''
        itemOld=item
    dt=time.time()-timeMark
    timeMark=time.time()
    fps=1/dt
    fpsFilter=.95*fpsFilter + .05 *fps
    cv2.putText(frame,str(round(fpsFilter,1))+'  fps  '+item+'   '+str(round(confidence,2)),(0,30),font,1,(0,0,255),2)
    cv2.imshow('nanoCam',frame)
    cv2.moveWindow('nanoCam',0,0)
    if cv2.waitKey(1)==ord('q'):
        break
cam.release()
cv2.destroyAllWindows()

import jetson.inference

import jetson.utils

import numpy as np

import time

import os

from gtts import gTTS

import threading

speak=True

item='Welcome to My Identify. Are you Ready to Rumble?'

confidence=0

itemOld=''

import cv2

print(cv2.__version__)

width=1280

height=720

flip=2

#Uncomment These next Two Line for Pi Camera

#camSet='nvarguscamerasrc ! video/x-raw(memory:NVMM), width=3264, height=2464, format=NV12, framerate=21/1 ! nvvidconv flip-method='+str(flip)+' ! video/x-raw, width='+str(dispW)+', height='+str(dispH)+', format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink'

#cam= cv2.VideoCapture(camSet)

def sayItem():

global speak

global item

while True:

if speak ==True:

output=gTTS(text=item, lang='en',slow=False)

output.save('output.mp3')

os.system('mpg123 output.mp3')

speak=False

x=threading.Thread(target=sayItem, daemon=True)

x.start()

#Or, if you have a WEB cam, uncomment the next line

#(If it does not work, try setting to '1' instead of '0')

cam=cv2.VideoCapture('/dev/video1')

cam.set(cv2.CAP_PROP_FRAME_WIDTH,width)

cam.set(cv2.CAP_PROP_FRAME_HEIGHT,height)

net=jetson.inference.imageNet('googlenet')

font=cv2.FONT_HERSHEY_SIMPLEX

timeMark=time.time()

fpsFilter=0

while True:

ret, frame = cam.read()

img=cv2.cvtColor(frame,cv2.COLOR_BGR2RGBA).astype(np.float32)

img=jetson.utils.cudaFromNumpy(img)

if speak==False:

classID, confidence = net.Classify(img,width,height)

if confidence>=.5:

item=net.GetClassDesc(classID)

if item!=itemOld:

speak=True

if confidence<.5:

item=''

itemOld=item

dt=time.time()-timeMark

timeMark=time.time()

fps=1/dt

fpsFilter=.95*fpsFilter + .05 *fps

cv2.putText(frame,str(round(fpsFilter,1))+' fps '+item+' '+str(round(confidence,2)),(0,30),font,1,(0,0,255),2)

cv2.imshow('nanoCam',frame)

cv2.moveWindow('nanoCam',0,0)

if cv2.waitKey(1)==ord('q'):

break

cam.release()

cv2.destroyAllWindows()

Jetson Nano

AI on the Jetson Nano LESSON 60: Make Your Nano Talk With Text to Speech

September 19, 2020 admin

In this lesson we show you how to teach the NVIDIA Jetson Nano to talk. We show two different ways. In the first, we use the espeak TTS engine. We then show how to do it with gTTS. We compare and contrast the pros and cons of each technique.

Jetson Nano

AI on the Jetson Nano LESSON 59: PWM on the GPIO Pins of the Jetson Nano

September 12, 2020 admin

In this lesson we show how to configure the GPIO pins on the Jetson Nano to produce PWM signals. We show how the PWM libraries can be run in python. The Jetson Nano can provide Pulse Width Modulation signals on two physical pins, pins 32 and 33. We take you through the step by step process in the video above.

Technology Tutorials

Tag Archives: NVIDIA

NVIDIA Jetson Orin Nano: Secret to Running Ollama on the GPU

The Problem: The “Canned” Installation

The “Secret Sauce” Solution

Step 1: Install the Nano Editor

Step 2: Create the Service Override

Step 3: Add the Configuration

Step 4: Reboot

Benchmarking the Results

AI on the Jetson Nano LESSON 62: Create a Streaming IP Camera from a Raspberry Pi Zero W

AI on the Jetson Nano LESSON 61: Image Recognition and Speech (TTS) on the Nano

AI on the Jetson Nano LESSON 60: Make Your Nano Talk With Text to Speech

AI on the Jetson Nano LESSON 59: PWM on the GPIO Pins of the Jetson Nano

Making The World a Better Place One High Tech Project at a Time. Enjoy!