Configuring the NVIDIA Jetson Thor

This is a work in progress as I try to get the Thor working and doing all the things.

Installing the OS

You can start with the Official User Guide

Then you should take a look at what containers are available for the Thor Here

Use a container rather than try to install things directly to save yourself the trouble of creating a dependency conflict mess.

Applications/Tools

Ollama

The latest Ollama supports Thor GPU inference just fine. There is one caveat when using large models and that if the Linux Buffer/Cache is consuming a large amount of memory, Ollama will not allocate the entire model into GPU(Cuda) memory and will end up running some layers on the CPU.

This is obviously undesirable. To get around this you can flush the cache buffers and force Ollama to NOT ev ict a model from memory.

Flush the buffers:

free && sync && echo 3 > /proc/sys/vm/drop_caches && free

Change the Ollama default configuration to not evict models

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
# Optional expose ollama to network
# Environment="OLLAMA_HOST=0.0.0.0:11434"
# Flash attention speeds up inference
Environment="OLLAMA_FLASH_ATTENTION=1"
# Adjust these settings to your needs 
# -1 means keep in ram forever ( unless another model needs to load )
# 0 means the model will be evicted immediatelly after use
# or 10m, 24h, etc to keep the model in memory for a certain time
Environment="OLLAMA_KEEP_ALIVE=-1"
# Check ollama documentation for these other ones
Environment="OLLAMA_MAX_LOADED_MODELS=1"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MAX_QUEUE=8"

YOLO

The easiest way to install YOLO is to use a prebuilt base container from NVIDIA:

I created a basic working yolo container with onnx export (for use with opencv) with the following Dockerfile.yolo

Build it:

docker build -t tools/yolo -f ./Dockerfile.yolo .

Run it:

# Notes: the default shm-size is to small for most jobs , I up mine to 4GB and that works fine.. you may find less or more can be used
docker run --gpus all -it --rm -v./:/workspace -v/data/datasets:/data/datasets --runtime nvidia --shm-size=4gb tools/yolo

# You can then run yolo commands like:
yolo detect train model=yolov8m data=/data/datasets/mydataset.yaml epochs=300 imgsz=640 device='0' augment=True batch=-1 degrees=90

OpenCV

Install dependancies

sudo apt-get update
sudo apt-get install -y build-essential cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
sudo apt-get install -y libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev python3.10-dev python3-numpy
sudo apt-get install -y libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libv4l-dev v4l-utils qv4l2
sudo apt-get install -y curl

Download the version you want:

https://github.com/opencv/opencv/tags/
wget https://github.com/opencv/opencv_contrib/tags/4.12.0.tar.gz opencv_contrib.tar.gz 

You need to have the correct CUDA GPU Capability set by

You can see the list here https://developer.nvidia.com/cuda-gpus

Build:

# Notes:  You need the latest 4.x branch ( Note 4.12 does not work without additional patches for Cuda13 4.13+ should workj once its out)  
# this should match the location of the contrib modules 
OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-4.12.0/modules
# SInce we are using the Jetson Thor, we need to disable the NVCUVID and NVCUVENC ( no support fot 
WITH_NVCUVID=OFF 
WITH_NVCUVENC=OFF
# 11.0 is the Jetson Thor GPU ( https://developer.nvidia.com/cuda-gpus )
CUDA_ARCH_BIN="11.0" 

cmake -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib-4.12.0/modules -D WITH_CUDA=ON -D WITH_CUDNN=ON -D CUDA_ARCH_BIN="11.0" -D CUDA_ARCH_PTX="" -D OPENCV_GENERATE_PKGCONFIG=ON -D WITH_GSTREAMER=ON -D WITH_LIBV4L=ON -D BUILD_opencv_python3=ON -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D WITH_NVCUVID=OFF -D WITH_NVCUVENC=OFF -D BUILD_EXAMPLES=ON -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local ..
make


You also need this patch:


sudo make install
echo 'export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH' >> ~/.bashrc
echo 'export PYTHONPATH=/usr/local/lib/python3.10/site-packages/:$PYTHONPATH' >> ~/.bashrc 

The trouble with torch

Honestly, I hate the python ecosystem and versioning… it’s a dependency rats nest, with conflicting documentation and blog posts.

Seriously, save yourself the trouble and use a container from Here

That said, if you really want to install natively:

First we need the NVPL Libraries

Version I installed:

wget https://developer.download.nvidia.com/compute/cuda/repos/debian12/sbsa/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get -y install nvpl

We need to get torch and torchvision installed from Jetson AI Lab. These must be installed in to a python virtual environment

These are the versions that worked for me

virtualenv torch  
source torch/bin/activate

wget https://pypi.jetson-ai-lab.io/sbsa/cu130/+f/bbe/d2ec4262e5dbd/torch-2.9.0-cp312-cp312-linux_aarch64.whl
wget https://pypi.jetson-ai-lab.io/sbsa/cu130/+f/c26/fb4d05c0d694c/torchvision-0.24.0-cp312-cp312-linux_aarch64.whl

pip install torch-2.9.0-cp312-cp312-linux_aarch64.whl
pip install torchvision-0.24.0-cp312-cp312-linux_aarch64.whl