How to pull a model with Q4_K_M quantization for balanced quality and size
Ollama installed and accessible from command line
What this does
Downloads a model that has been quantized with Q4_K_M format, compressing weights to roughly one-third of full precision while retaining most output quality. After this guide the quantized model will be ready for inference on mid-range hardware.
Steps
Pull the Q4_K_M variant of a model. Append
:q4_k_mto the model name to request the quantized version.ollama pull llama3.2:q4_k_mExpected output: Progress bars followed by
success.Verify the quantization level in the model metadata. Confirms the installed variant matches the requested tag.
ollama show llama3.2:q4_k_mExpected output: Metadata including the quantization type and parameter count.
Check disk usage for the quantized file. Q4_K_M files are typically 30-40% smaller than the full-precision variant.
ollama list | grep q4_k_mExpected output: A row with the model name and its size in the SIZE column.
- Record the local run evidence. Save the exact command, runtime or package version, model name if applicable, and observed output so the result can be reproduced later.
Verification
ollama show llama3.2:q4_k_m | grep -i quant
# Expected: a line showing "quantization: q4_k_m" or similar
Common failures
model not found- The requested model does not have a Q4_K_M variant in the library; try:q4_0or:q4_k_sas alternatives.disk full during download- The download requires temporary space in addition to the final file; free at least 2 GB extra.incomplete download- Network interruption; re-runollama pullto resume from the last checkpoint.confusing Q4_K_S with Q4_K_M- Q4_K_S is smaller but lower quality; verify the tag after pull withollama list.
Operator checkpoint
Before treating this as solved, write down the local runtime, model or package version, hardware/backend if relevant, and the verification output. This keeps the guide useful as a Will-It-Run style decision instead of a one-off command transcript.