BENCHMARK · MBPP-PLUS
Coding

MBPP+ (EvalPlus)

EvalPlus's augmented MBPP suite: introductory Python programming tasks with stronger hidden tests than the original MBPP benchmark. Measures short-form functional code generation quality.

Test runs
0
Metric
pass@1
Range
/100
Best score
—
+Submit a scoreRunner script →Original source →Vendor leaderboard →JSON →
METHODOLOGY

How we run MBPP+ at runlocalai

  1. Spin up the target model on its native runtime (Ollama / vLLM / llama.cpp) at the listed quantization on the listed hardware.
  2. Generate one deterministic sample per task through the local OpenAI-compatible endpoint.
  3. Score the JSONL samples with the official EvalPlus evaluator: python -m evalplus.evaluate mbpp --samples $SAMPLES.
  4. Score = pass@1 on MBPP+ x 100.
  5. Publish the raw generation log, official scorer log, sanitized samples, and raw completions to a public GitHub Gist before the row enters the leaderboard.

Runner source: scripts/run-humaneval-plus.ts and scripts/evalplus_openai_generate.py in the public repo.

HOW TO READ THE SCORE

Score is pass@1 percentage on the EvalPlus-extended MBPP test suite. 100 = every task solved correctly. MBPP+ is usually easier than HumanEval+ for strong coding models, but the augmented tests catch many brittle or partial solutions.

LEADERBOARD

Public reviewed runs, ranked

0 rows
NO RUNS YET

No test runs have been submitted for this benchmark yet. If you have rig time, see /benchmarks/submit for the contribution flow.