RUNLOCALAIv38
->Will it run?Best GPUCompareTroubleshootStartLearnPulseModelsHardwareToolsBench
Run check
RUNLOCALAI

Independently operated catalog for local-AI hardware and software. Hand-written verdicts. Source-cited claims. Reproducible commands when we have them.

OP·Eruo Fredoline
DIR
  • Models
  • Hardware
  • Tools
  • Benchmarks
TOOLS
  • Will it run?
  • Compare hardware
  • Cost vs cloud
  • Choose my GPU
  • Prompting kits
  • Quick answers
REF
  • All buyer guides
  • Learn local AI
  • Methodology
  • Glossary
  • Errors KB
  • Trust
  • Suggest a feature
EDITOR
  • About
  • Author
  • How we make money
  • Editorial policy
  • Contact
LEGAL
  • Privacy
  • Terms
  • Sitemap
MAIL · MONTHLY DIGEST
Get monthly local AI changes
Monthly recap. No spam.
DISCLOSURE

Some links on this site are affiliate links (Amazon Associates and other first-class retailers). When you buy through them, we earn a small commission at no extra cost to you. Affiliate links do not influence our verdicts — there are cards we rate highly that we don't have affiliate relationships with, and cards that sell well that we refuse to recommend. Read more →

© 2026 runlocalai.coIndependently operated
RUNLOCALAI · v38
  1. >
  2. Home
  3. /Learn
  4. /How-to
  5. /How to quantize a model for llama.cpp
HOW-TO · SET

How to quantize a model for llama.cpp

advanced·30 min·By Eruo Fredoline
Target environment
Ubuntu 24.04 · Ollama 0.4.xWindows 11 · Ollama 0.4.xmacOS 15 · Ollama 0.4.x
PREREQUISITES

llama.cpp compiled with quantization support, base model in FP16 format

What this does

Converts a full-precision model (FP16) into a quantized GGUF format, reducing file size and memory footprint at the cost of some accuracy. The resulting file runs efficiently on consumer hardware.

Steps

  1. Identify the model type for quantization. Run the help command to list supported types.

    ./llama-quantize --help

    Expected output: List of available quantization types such as Q4_K_M, Q5_K_S, Q8_0.

  2. Run the quantize binary on the FP16 model file. Specify the source file, destination file, and the target quantization type.

    ./llama-quantize /path/to/model-fp16.gguf /path/to/model-Q4_K_M.gguf Q4_K_M

    Expected output: Progress bar showing layer-by-layer conversion and final file size comparison.

  3. Verify the output file is a valid GGUF with reduced size. The new file should be 3-6x smaller than the FP16 source.

    ls -lh /path/to/model-Q4_K_M.gguf

    Expected output: File size significantly smaller than the FP16 original, typically under 8 GB for a 7B model at Q4_K_M.

  • Record the local run evidence. Save the exact command, runtime or package version, model name if applicable, and observed output so the result can be reproduced later.

Verification

./llama-quantize --dry-run /path/to/model-Q4_K_M.gguf
# Expected: dry-run output confirming quantization type and layer stats without error

Common failures

  • Unsupported quantization type selected — The model architecture does not support the chosen type. Use a lower quantization level such as Q8_0.
  • Out of disk space during conversion — Temporary files require space equal to the input file. Free sufficient space or work from a directory with adequate storage.
  • Model not in GGUF format — Convert raw model files to GGUF first using convert.py in the llama.cpp repository before quantization.
  • Quantized model produces garbled output — Aggressive quantization (below Q4) may degrade output quality. Retry with Q5_K_S or Q4_K_M.
  • Quantization hangs at a specific layer — Interrupt and restart with a less aggressive quantization type.

Related guides

  • How to run inference with llama.cpp server
  • How to use llama.cpp with CUDA acceleration
  • Course Local AI Fundamentals
RELATED GUIDES
SET
How to use llama.cpp with CUDA acceleration
SET
How to run inference with llama.cpp server
← All how-to guidesCourses →