How do you fix "Slow tokens/sec on capable GPU (silent CPU fallback)"?

**1. Confirm the GPU is being used:** ```bash # Watch GPU usage while a prompt runs watch -n 0.5 nvidia-smi ``` If utilization stays at 0%, you're on CPU. **2. PyTorch — check the install:** ```python import torch print(torch.cuda.is_available(), torch.version.cuda) ``` If `False` / `None`, reinstall with the CUDA index: ```bash pip uninstall torch torchvision -y pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124 ``` **3. llama.cpp — set GPU layers:** ```bash # Offload all layers to GPU; -1 = all ./llama-cli -m model.gguf -ngl 99 -p "test" ``` Watch the loader output — it should say `offloaded N/N layers to GPU`. **4. Ollama — check env:** ```bash env | grep OLLAMA # Unset any blockers unset OLLAMA_NO_GPU ollama serve ``` **5. Confirm you built with the right backend.** A llama.cpp built without `GGML_CUDA=1` silently runs on CPU even with `-ngl` set. Rebuild with the correct backend.

Configuration

Verified by owner

Slow tokens/sec on capable GPU (silent CPU fallback)

(no error — output is correct but tok/s is 5-10× slower than expected)

By Eruo Fredoline · Last verified Oct 7, 2026

Cause

The model loaded but is running on CPU instead of GPU. Common in Python: pip install torch without specifying CUDA picks up the CPU-only wheel; in Ollama: OLLAMA_NO_GPU=1 was set; in llama.cpp: -ngl 0 was passed (or omitted on a build that doesn't auto-offload).

Symptom: nvidia-smi shows 0% GPU utilization while inference runs. Per-token latency is dominated by CPU memory bandwidth, which is ~10× slower than GPU VRAM.

Solution

1. Confirm the GPU is being used:

# Watch GPU usage while a prompt runs
watch -n 0.5 nvidia-smi

If utilization stays at 0%, you're on CPU.

2. PyTorch — check the install:

import torch
print(torch.cuda.is_available(), torch.version.cuda)

If False / None, reinstall with the CUDA index:

pip uninstall torch torchvision -y
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

3. llama.cpp — set GPU layers:

# Offload all layers to GPU; -1 = all
./llama-cli -m model.gguf -ngl 99 -p "test"

Watch the loader output — it should say offloaded N/N layers to GPU.

4. Ollama — check env:

env | grep OLLAMA
# Unset any blockers
unset OLLAMA_NO_GPU
ollama serve

5. Confirm you built with the right backend. A llama.cpp built without GGML_CUDA=1 silently runs on CPU even with -ngl set. Rebuild with the correct backend.

Cloud GPU rental

Not enough VRAM for this model? Rent a GPU by the hour

If tokens are slow because part of the model runs on the CPU (it does not fit in your VRAM), you can rent a GPU with enough VRAM for every layer by the hour.

RunPod

billed per GPU-hour

Hourly GPU pods, in community-cloud and secure-cloud tiers.

The two tiers run on different hosts and terms. Check which tier a pod is on before a long job.

Rent on RunPod

Vast.ai

billed per GPU-hour

Marketplace of host-listed GPUs; price and reliability vary by host.

Host quality varies. Filter for verified hosts and check the host's reliability before a long run.

Rent on Vast.ai

Referral links: RunLocalAI may earn a commission or account credits if you sign up and rent, at no extra cost to you. Hourly rates and which GPUs are listed change all the time, so check the provider's page before you start. Editorial opinions are independent of the referral relationship; see how we make money.

Related errors

Did this fix it?

If your case was different, email Contact support with what you saw and we'll update the page. If it worked but took different commands on your platform, we want to know that too.