CPU Offload
Definition pending
We've cataloged "CPU Offload" but haven't written a full definition yet. Definitions are hand-curated rather than auto-generated, so it takes time to cover every term.
Want this one prioritized? Email us and we'll bump it.
Practical example
CPU offload spills model layers from GPU VRAM to system RAM when you run out of VRAM. A 70B Q4 model needs ~40 GB of memory. If you have 24 GB VRAM + 64 GB RAM, llama.cpp puts ~24 GB of layers on GPU, the remaining ~16 GB on CPU. Your inference speed drops from ~30 tok/s to ~3 tok/s — a 10× penalty.
Workflow example
Configure CPU offload in Ollama: set the number of GPU layers in the Modelfile with PARAMETER num_gpu 20 (for 20 layers on GPU, rest on CPU). Find the sweet spot: ollama run --verbose shows layer assignment. Typical split for 70B on 24 GB: ~30 layers GPU, ~50 layers CPU = 15–20 tok/s GPU + 3–5 tok/s CPU = ~8 tok/s effective. If speed is unacceptable, use a smaller model or get more VRAM.