pass@1

pass@1 is the probability that a model's first generated solution passes the unit tests for a coding problem, computed from a single sample at temperature 0 (or pass@k with k=1 estimated from larger samples). Used in HumanEval, MBPP, LiveCodeBench, and most coding benchmarks.

Distinct from pass@10 or pass@100, which let the model take multiple tries — those measure capability under best-of-N scaling, while pass@1 measures the model's "first guess" reliability.

For local AI, pass@1 is what users actually experience: when you ask a model for code, you usually run the first thing it gives you. Benchmark results that quote pass@10 or pass@100 don't translate to single-shot use.

An operator evaluating two local coding models — say a Qwen2.5-Coder 7B versus DeepSeek-Coder 7B — for an autocomplete tool should run each with a single greedy (temperature 0) sample per HumanEval problem and compare pass@1, since that mirrors how the tool actually behaves: it generates one completion and the user either accepts or rejects it. Reporting pass@10 instead would flatter both models by letting them retry, which doesn't reflect the deployed experience. If a vendor's marketing page quotes a suspiciously high number, check whether it's pass@1 or pass@100 — the gap between the two is often 15-20+ points on the same model, and conflating them makes weaker models look competitive with stronger ones.