GGUF

GGUF (GGML Unified Format) is the file format used by llama.cpp and its ecosystem (Ollama, KoboldCPP, LM Studio). A single file contains the quantized model weights, tokenizer, and metadata — no separate config files needed. Replaced the older GGML format in late 2023.

Quantization variants live as suffixes: Q4_K_M, Q5_K_M, Q8_0, F16, etc. The K-quants (Q4_K_M, Q5_K_M) are mixed-precision — different layers get different bit widths based on sensitivity. Q4_K_M, despite the name, averages 4.83 bits per parameter because attention layers stay at 6-bit.

GGUF is single-file, mmap-friendly (the OS pages model weights as needed instead of loading everything upfront), and runs on every platform llama.cpp supports — including phones and Raspberry Pis. The file extension is universal: model.Q4_K_M.gguf.

An operator wants to run Llama 3.1 8B on a Raspberry Pi 5 for an offline home-automation assistant — no GPU, 8GB RAM, no internet after setup. They pull Llama-3.1-8B-Instruct-Q4_K_M.gguf, a single ~4.9GB file with weights, tokenizer, and chat template metadata baked in, and point llama.cpp at it directly — no Python environment, no separate config.json or tokenizer.json to keep in sync. Because GGUF is mmap-friendly, llama.cpp doesn't need to load the full 4.9GB into RAM upfront; it pages weights in as layers execute, which is what makes the model runnable at all on a device with limited memory. The same file, unmodified, also runs on their Mac laptop via LM Studio and their Windows desktop via Ollama — the portability is the entire reason GGUF displaced GGML as the llama.cpp-ecosystem standard.