GSM8K

GSM8K is a benchmark of 8,500 grade-school math word problems requiring 2–8 reasoning steps. Models are scored by whether the final numeric answer matches the ground truth. Designed by OpenAI in 2021 to test multi-step arithmetic reasoning.

Long since saturated by frontier models (>95%) but still a useful local-AI sanity check: a quantization that drops GSM8K by 5+ points has lost reasoning fidelity, even if perplexity barely moved.

Common gotchas: chain-of-thought prompting boosts GSM8K dramatically (often +20 points), so benchmark numbers are only comparable when the prompting strategy matches.

Suppose you quantize Llama 3.1 8B from FP16 down to Q3_K_M to fit a tighter VRAM budget and want to confirm the model still reasons correctly, not just that perplexity looks stable. Running the full MMLU suite takes too long on a single GPU, so you instead run a 200-problem GSM8K subset with consistent zero-shot prompting before and after quantization. A 3-4 point drop is typical noise from aggressive quantization; anything beyond 8-10 points signals the quant scheme is clipping activations that matter for multi-step arithmetic, and you should fall back to Q4_K_M or an importance-matrix (imatrix) quant instead. This kind of before/after diff is far more informative locally than chasing a single absolute score, since GSM8K is now near-saturated for frontier models anyway.