BLK · CODING BENCHMARKSfirst-party · per-quant · reproducible

Coding benchmark leaderboard

Reviewed HumanEval+ scores for open-weight local models at real-world quantizations. First-party rows were run by RunLocalAI; community rows render only after review. Every public row links to its raw test-run Gist.

Test runs
0
Unique models
0
Quant variants
0
Benchmark suites
0
METHODOLOGY

How these numbers are produced

  1. Each model loads into the listed runtime (Ollama / vLLM / llama.cpp) at the listed quantization on the listed hardware.
  2. EvalPlus is used as the official scorer. For OpenAI-compatible local runtimes, we first generate deterministic JSONL samples, then score them with python -m evalplus.evaluate.
  3. Greedy sampling (deterministic; one sample per problem). Score = pass@1 on HumanEval+ (164 problems with augmented tests) × 100.
  4. Raw stdout + stderr captured and posted to a public GitHub Gist before the row enters this table. Click any score to verify.
  5. Runner source: scripts/run-humaneval-plus.ts (public repo) — reproducible end-to-end.
NO RUNS YET

We’re actively running HumanEval+ across the catalog. Check back this week — every kitted model on /prompting is on the queue.

A note on quant comparisons. The same model at different quantization levels can score materially differently — especially on code, where precision loss hurts. When you see Q4_K_M next to Q6_K for the same model, the gap is the “quant tax” on coding specifically. Take that into account when picking what to run on /will-it-run.