MMLU

MMLU (Massive Multitask Language Understanding) is a benchmark that tests a language model's knowledge across 57 subjects, from STEM to humanities. Models answer multiple-choice questions; the score is the percentage correct. It is widely used to compare model performance, and operators see MMLU scores in model cards on Hugging Face or in Ollama model descriptions. A higher MMLU score suggests broader factual knowledge, but it does not measure reasoning, safety, or instruction-following.

MMLU was introduced in 2020 by Hendrycks et al. to evaluate models on a diverse set of tasks, covering elementary mathematics, US history, computer science, law, and more. Each subject has a set of questions with four answer choices. Models are typically evaluated in a few-shot setting (e.g., 5 examples per subject) to condition the model on the task format. The benchmark is challenging because it requires both broad knowledge and the ability to select the correct answer from similar options. While MMLU is a standard reference, it has limitations: it is multiple-choice, so it does not test generation or open-ended reasoning; it can be contaminated if training data includes test questions; and it may not reflect real-world usage where models need to follow instructions or handle ambiguity. Operators should interpret MMLU scores alongside other benchmarks like HumanEval (coding) or GSM8K (math) for a fuller picture.

When comparing Llama 3.1 8B and Mistral 7B, an operator might see MMLU scores of 66% vs 62% on a model card. This suggests Llama 3.1 8B has slightly broader factual knowledge. However, if the operator's use case is creative writing, MMLU is less relevant than benchmarks like MT-Bench or Chatbot Arena rankings.

On Hugging Face, model cards for Llama, Mistral, or Qwen include MMLU scores under 'Evaluation Results.' In Ollama, ollama show llama3.1:8b prints the model's MMLU score if the maintainer included it. Operators can also run the MMLU benchmark themselves using the lm-evaluation-harness library: lm_eval --model hf --model_args pretrained=meta-llama/Llama-3.1-8B --tasks mmlu.