TurkishMMLU (Generative)
Turkish-translated Massive Multitask Language Understanding benchmark: 900 questions across 9 subjects (Biology, Chemistry, Geography, History, Mathematics, Philosophy, Physics, Religion & Ethics, Turkish Language & Literature). We use the generative variant: the model emits a letter (A-E) as text and we parse it. Comparable to other generative MMLU runs.
How we run TurkishMMLU at runlocalai
- Load the AYueksel/TurkishMMLU dataset from HuggingFace.
- For each of 9 subtasks, build 5-shot prompts using the dev split as exemplars.
- Send each test question via the OpenAI-compatible chat completions endpoint (Ollama / vLLM / etc.).
- Parse the model's response for the first standalone letter A-E.
- Aggregate accuracy per subtask and overall.
Why generative, not loglikelihood: The traditional lm-evaluation-harness approach uses loglikelihood scoring, which requires the inference backend to return token logprobs. Ollama's chat/completions endpoint does not return logprobs, so we use a generative letter-pick approach. Generative MMLU scores typically land within ~3pp of loglikelihood MMLU for well-instructed models.
Runner source: scripts/run-turkish-mmlu.ts and scripts/turkish_mmlu_generative.py in the public repo.
Score is overall accuracy percentage across all 900 questions. Random baseline is 20% (5 answer choices). Strong models score 40-60%; weak/broken setups score below 20% (suggests context overflow or chat template mismatch).
Public reviewed runs, ranked
No test runs have been submitted for this benchmark yet. If you have rig time, see /benchmarks/submit for the contribution flow.