llama.cpp

What llama.cpp actually is

llama.cpp is the single most important piece of code in the local-AI ecosystem — a header-light C/C++ inference engine for transformer models with hand-rolled SIMD kernels for every CPU instruction set people care about, plus CUDA, Metal, ROCm, Vulkan, SYCL, and CANN backends maintained in-tree. It is the engine that Ollama, LM Studio, llama-cpp-python, GPT4All, and most local-AI desktop apps in the last three years have been built on top of.

It is also the project that defined the GGUF quantization format — the only format with truly cross-runtime, cross-OS, cross-GPU portability. See /systems/quantization-formats.

Where it fits in the stack

llama.cpp lives at the engine layer. It is what you reach for when:

• You want maximum control over how the model loads, where layers go, and what kernels run
• You're integrating local inference into a larger application and don't want a separate HTTP daemon
• You're targeting hardware no other engine supports (older CPUs, Vulkan-only GPUs, embedded boards)
• You're benchmarking and need reproducible, deterministic, well-instrumented runs

Most users will never touch llama.cpp directly — they'll touch Ollama or LM Studio, which wrap it. But every operator who ships local-AI to production should at least understand what the wrapper is hiding, because every llama.cpp tunable maps directly to a knob the wrapper either exposes or buries.

Best use cases

• Embedded / edge inference. llama.cpp's static binary, no-Python footprint, and SIMD-only fallback path make it the only realistic option for Jetson, Raspberry Pi, x86 industrial boxes, etc.
• Low-level benchmarking. llama-bench is the canonical tool for reproducible local-AI throughput numbers — every benchmark in our /compatibility matrix that uses GGUF is generated through it.
• Custom build into another binary. llama.cpp compiles cleanly as a static lib; if you're shipping a Tauri / Electron / native app with bundled inference, this is your engine.
• Speculative decoding research. llama.cpp's speculative-sampling implementation is well-documented and easy to instrument.
• CPU-only / hybrid CPU-GPU offload. No other engine in 2026 splits across CPU and GPU layers with the same finesse.

OS support

Hardware / backend support

The most comprehensive backend list of any local-AI engine in 2026:

• CUDA (NVIDIA, all consumer + datacenter cards)
• Metal (Apple Silicon)
• ROCm / HIP (AMD; see ROCm)
• Vulkan (cross-vendor GPU compute — Intel Arc, AMD on Windows, older NVIDIA)
• SYCL (Intel Arc / Intel iGPU)
• CANN (Huawei Ascend)
• CPU SIMD: AVX-512, AVX2, AVX, NEON, plus pure-scalar fallback

For multi-GPU, llama.cpp supports layer-split (per-layer device assignment) and row-split (tensor-parallel per layer). Layer-split is what wrappers default to; row-split needs explicit flags and is a real win on dual-3090 builds. See /guides/running-local-ai-on-multiple-gpus-2026.

Model / quant format support

GGUF is llama.cpp's native format. The full ladder:

• F32 / F16 / BF16 — reference; no quality loss; for benchmark baselines
• Q8_0 — near-lossless; ~2× memory savings vs F16
• Q6_K, Q5_K_M, Q5_K_S, Q4_K_M, Q4_K_S — production sweet spot
• Q3_K_M, Q3_K_S, Q2_K — extreme compression; quality drop matters on coding / math tasks
• IQ-series (IQ4_NL, IQ3_M, IQ2_S, etc.) — codebook-based quants; better quality at a given size, slower kernels

Conversion of HF safetensors → GGUF is done with the in-tree convert_hf_to_gguf.py. Quantization to a target format is done with llama-quantize. Both are well-tested for Llama / Qwen / Mistral / DeepSeek lineage; exotic architectures may need a kernel landed in main first.

Setup path

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON   # or -DGGML_METAL=ON, -DGGML_HIPBLAS=ON, etc.
cmake --build build --config Release -j
./build/bin/llama-cli -m model.gguf -p "Hello" -n 64

For an HTTP server: ./build/bin/llama-server -m model.gguf --port 8080. For benchmark sweeps: ./build/bin/llama-bench -m model.gguf -p 0 -n 128.

What breaks first

1. CUDA arch flag mismatches. -DCMAKE_CUDA_ARCHITECTURES=86 (3090) vs 89 (4090) vs 90 (H100) — building with the wrong arch produces a binary that loads but uses CPU silently.
2. GGUF file truncation on download. Hugging Face occasionally serves partial files; verify SHA256 if outputs look garbled.
3. GPU offload greater than VRAM. -ngl 999 is a footgun on 12 GB cards; the OOM error often comes from the KV cache, not the weights.
4. Custom chat templates. llama.cpp ships a Jinja-template engine for chat formatting; missing or wrong templates produce subtly broken outputs.
5. ROCm version drift on Linux. ROCm 6.x vs 7.x build flags differ; see ROCm device error.

Alternatives by intent

Best pairings

• llama-cpp-python — embed the same engine in a Python service
• Open WebUI — point at llama-server's OAI-compatible endpoint
• LangChain / LlamaIndex — the Llama.cpp connector wraps either the binary or the Python binding
• Continue.dev — same OAI-compatible endpoint pattern as Ollama

Who should avoid llama.cpp

• Anyone unwilling to compile. Use Ollama — same engine, no cmake.
• Multi-tenant production serving with continuous batching. llama.cpp supports parallel decoding but is not architected for the throughput class vLLM hits.
• Workloads on H100 / H200 where FP8 transformer-engine kernels matter. Use TensorRT-LLM.

Related

• Stacks: /stacks/local-coding-agent, /stacks/dual-3090-workstation
• System guides: /systems/quantization-formats, /guides/running-local-ai-on-multiple-gpus-2026
• Hardware: RTX 3090, Apple M3 Ultra, RX 7900 XTX
• Errors: /errors/rocm-device-not-found, /errors/wsl2-gpu-not-detected

Install via package manager: macOS brew install llama.cpp, or build from source: git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && cmake -B build && cmake --build build --config Release. The server binary is at ./build/bin/llama-server. Download a GGUF model from HuggingFace (e.g. hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF). Start: ./build/bin/llama-server -m models/Llama-3.2-3B-Instruct-Q4_K_M.gguf --port 8080. Verify: curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"messages":[{"role":"user","content":"Hello"}]}'. The server exposes OpenAI-compatible /v1/chat/completions and /v1/completions endpoints plus a web UI at http://localhost:8080. First run time is download-only — inference starts instantly after model load (~5–30 seconds depending on model size). llama.cpp runs on CPU, CUDA, Metal (Apple Silicon), Vulkan, SYCL (Intel), and ROCm (AMD) backends — specify via -ngl 99 to offload all layers to GPU. Time-to-first-response from zero: ~1 minute including model download for a 3B GGUF.

Best for: single-user local chat and inference across every hardware target (CPU, GPU, Apple Silicon), embedding generation with broad model support, GGUF-first model ecosystems, CPU-only server deployments where NVIDIA GPUs aren't available, speculative-drafting pipelines with small/large model pairs, offline/air-gapped inference where Docker or complex Python environments are unavailable. Not suited for: high-concurrency production API serving (>10 concurrent requests — use vLLM), latency-competitive deployments where every millisecond counts (use TensorRT-LLM), researcher workflows that need to load models from HuggingFace safetensors directly without GGUF conversion, Windows-first deployments (WSL2 is the supported path).

Use llama.cpp for maximum hardware coverage — it is the only production-quality inference engine that runs on CPU, Apple Silicon, AMD Vulkan, Intel SYCL, and NVIDIA CUDA with a single codebase. The GGUF format ecosystem is the widest: tens of thousands of pre-quantized models on HuggingFace at every quant level from Q2 to Q8. Switch to vLLM when you need multi-tenant production serving on NVIDIA datacenter GPUs — llama.cpp's throughput per-GPU is 3–10× lower than vLLM. Use MLX-LM when on Apple Silicon and you want Apple-optimized inference with Swift-friendly tooling; llama.cpp on Metal is competitive but MLX-LM often edges it on memory bandwidth utilization. Use Ollama when you want a polished CLI and model management layer — Ollama wraps llama.cpp as its backend but adds download/versioning/conversation management that raw llama.cpp lacks. Stick with llama.cpp for embedding generation, speculative decoding, and any CPU-only deployment.

Problem: llama_model_load: error loading model: invalid model file. Fix: The GGUF file is corrupted or incompatible with your llama.cpp version. Verify with ./build/bin/quantize --validate models/your-model.gguf. Re-download from the bartowski or lmstudio-community quantization on HuggingFace. Problem: GPU offloading slower than CPU-only on Apple Silicon. Fix: Metal backend may be shader-compiling on first inference; run a warmup request first. Ensure -ngl 99 (not -ngl 0) and check that Metal is the active backend with -v flag. Some models have architecture variants (e.g. DeepSeek MoE) that require specific llama.cpp build flags — rebuild with -DGGML_CUDA=ON, -DGGML_METAL=ON, or equivalent. Problem: Server crashes on >32K context. Fix: llama.cpp defaults to model-native context window. Override with -c 8192 if your hardware can't handle full context. Flash attention (-fa) reduces KV-cache memory by ~30% — enable it for long-context workloads.