How to Use the MMLU Benchmark, Step by Step

How to use MMLU benchmark runs well comes down to five moves: decide what question the score should answer, pick the MMLU variant that matches your model, run it with a standard harness such as EleutherAI's lm-evaluation-harness or Inspect, check a handful of raw answers before the full run, and report the score with its settings and error bars. In 2026 MMLU is best used to catch regressions in your own models (after fine tuning, quantization or distillation) and to test small or multilingual models. It is a poor way to rank frontier models, because they sit near its ceiling.

This guide is the hands on companion to our explainer on LLM benchmarks, which covers what MMLU is. Here we focus on running it and reading the result.

Step 1: Decide what the score is for

Before you install anything, write down the decision the number will inform. Three common cases:

• Regression check. You fine tuned, merged or quantized a model and want to know whether it lost general knowledge. MMLU is still good at this, because you compare your model against itself under identical settings.
• Picking a small or open weight model. MMLU still separates many smaller models, but treat it as one signal among several, next to task tests like the ones in our guide on how to evaluate LLM model performance.
• Ranking frontier models. Skip it. Use harder tests such as MMLU-Pro or GPQA instead, and see MMLU vs MMLU Pro and MMLU vs GPQA for why.

If you only need someone else's number, you do not need to run anything. You do still need Step 6, because published MMLU numbers are often not comparable.

Step 2: Know what you are running

MMLU comes from a 2020 paper by Dan Hendrycks and colleagues, published at ICLR 2021 (paper). It covers 57 subjects, from elementary mathematics and US history to law and medicine, with four answer options per question, so random guessing scores 25 percent.

The standard copy is the cais/mmlu dataset on Hugging Face, released under the MIT license and not gated (dataset). Its splits matter for how you run it:

Subjects are uneven. Professional law has 1,534 test questions, while many subjects, such as college chemistry, have exactly 100. Keep that in mind when you read per subject scores later.

Step 3: Pick the variant that fits your model

"MMLU" now names a family of tasks. lm-evaluation-harness alone ships mmlu, mmlu_continuation, mmlu_generative, Flan style chain of thought variants, mmlu-redux, mmlu_pro and the translated openai-mmmlu, among others (MMLU task docs). The right choice depends on how your model can be scored.

Two rules of thumb follow from the harness README. Models that do not return log probabilities can only run generation tasks, while local models and APIs that return prompt log probabilities can run every task type (lm-evaluation-harness). And the Inspect MMLU docs recommend pretrained models over instruction tuned ones for the five shot prompt, because some chat models try to talk through the problem instead of answering with a letter.

Step 4: Install and smoke test

For a local model with lm-evaluation-harness (version 0.4.13 on PyPI as of October 2026), install the backend you need. Since December 2025 the base package no longer bundles transformers or torch:

pip install "lm_eval[hf]"        # or lm_eval[vllm], lm_eval[api]

Then run a tiny sample with logged outputs:

lm_eval --model hf \
  --model_args pretrained=YOUR_ORG/YOUR_MODEL \
  --tasks mmlu \
  --num_fewshot 5 \
  --limit 10 \
  --batch_size auto \
  --log_samples \
  --output_path results/smoke

Open the logged samples and read a few prompts and answers by eye. You are checking three things: the prompt looks like "question, A to D choices, Answer:", the few shot examples come from the right subject, and nothing is cut off. The harness docs say --limit is for testing only, so never report a limited run as an MMLU score.

For chat or API models, the harness README makes the same suggestion: look at sample outputs with --limit 10 first to confirm answer extraction works. This step catches the most expensive mistake in MMLU runs. In the Inspect maintainers' own test runs, Gemini 3 Pro scored 0 percent because it talked through each problem instead of giving the letter the scorer expected. That says nothing about its knowledge and everything about formatting.

Step 5: Run the full benchmark

Once the smoke test looks right, drop --limit and run the full set:

lm_eval --model vllm \
  --model_args pretrained=YOUR_ORG/YOUR_MODEL \
  --tasks mmlu \
  --num_fewshot 5 \
  --batch_size auto \
  --output_path results/full

A few settings deserve care:

• Shots. The default MMLU config draws few shot examples from the dev split in a fixed order (first_n), but it does not set the count, so pass --num_fewshot 5 to match the paper's usual five shot setup.
• Chat template. For instruction tuned models, --apply_chat_template wraps prompts in the model's chat format and turns few shot examples into a multi turn conversation by default. Scores with and without it are different numbers.
• Reasoning models. The harness supports stripping thinking traces with a think_end_token model argument. Inspect removes its answer token cap when you configure reasoning effort, after a May 2026 fix for empty answers.
• Subsets. To check one area, run mmlu_stem, mmlu_humanities, mmlu_social_sciences or mmlu_other, or a single subject such as mmlu_anatomy.

For an API model through Inspect, the equivalent is:

pip install inspect-evals
inspect eval inspect_evals/mmlu_0_shot --model openai/YOUR_MODEL
inspect eval inspect_evals/mmlu_5_shot --model openai/YOUR_MODEL -T subjects=anatomy,astronomy

Inspect's zero shot task uses a Simple Evals style prompt and accepts -T cot=true for chain of thought and -T language=FR_FR for the translated MMMLU set, which covers 14 languages.

Step 6: Read the result like a reviewer

The harness prints a table with accuracy and a standard error for every subject and group. Four checks keep you honest.

Weighting. The harness's mmlu group weights subjects by their number of questions, so professional law counts about 15 times as much as college chemistry. Some papers report a plain average across the 57 subjects instead. Say which one you used.

Noise. Error bars depend on sample size. As a rough guide (our arithmetic, using the binomial standard error): at 70 percent accuracy, a 100 question subject has a standard error of about 4.6 points, while the full 14,042 question test has about 0.4 points. A five point swing in one small subject can be noise; a two point drop in the overall score after quantization usually is not.

Implementation. The same model can get very different "MMLU scores" from different code. In a June 2023 investigation, Hugging Face ran LLaMA 65B through three MMLU implementations and got 0.637 with HELM, 0.636 with the original code and 0.488 with the lm-evaluation-harness version of that time, and the model rankings changed between implementations (Hugging Face blog). The harness later switched to the original implementation. The lesson stands: compare only numbers produced with the same code, prompt and shot count. HELM publishes its own MMLU runs, which are useful when you want a third party baseline.

Label errors. MMLU has known flaws. The MMLU-Redux team had 14 experts relabel 5,700 questions and estimated that 6.49 percent of all MMLU questions contain errors; in the virology subset, 57 percent of the questions they checked had problems (paper). The Inspect maintainers also found 105 duplicated rows in the test split: 78 questions listed under two subjects and 27 exact repeats. If a model's gains are concentrated in error heavy subjects, be suspicious, and consider running mmlu_redux_generative alongside the full set.

A simple reporting template

When you share an MMLU number, include every item below so someone else can reproduce it:

1. Task name and variant (for example mmlu, five shot, log probability scoring).
2. Harness and version (for example lm-evaluation-harness 0.4.13), plus the dataset revision if you pinned one.
3. Model revision, precision or quantization, and whether a chat template was applied.
4. Overall accuracy with its standard error, and whether the average is weighted by subject size.
5. Any subjects you excluded and why.

That list takes two minutes to write and saves hours of arguments about why your number differs from a model card.

When to use something else

MMLU tests recall of exam style knowledge. It does not test coding, tool use or long tasks. For software work, look at SWE-bench and our MMLU vs SWE-bench comparison. For broad reasoning across many unusual tasks, BIG-bench is an older but wider suite. For production apps, your own test set beats any public benchmark; our LLM evaluation frameworks guide compares the tools for that.

FAQ

Is MMLU run zero shot or five shot?

Both are common. The original setup is five shot, with five examples per subject taken from the dev split, and that remains the usual choice for base models. Instruction tuned and API models are often run zero shot with a direct answer format, as in Inspect's mmlu_0_shot task. The two settings give different numbers, so always state which one you used.

Can I run MMLU on GPT or Claude through an API?

Yes. Use a generation task, such as mmlu_generative in lm-evaluation-harness or mmlu_0_shot in Inspect, since most chat APIs do not return the log probabilities the original scoring needs. Test ten samples first to confirm the model answers with a letter, and expect results to differ from log probability scores on the same model.

How many questions does a full MMLU run use?

The test split has 14,042 questions across 57 subjects. Inspect's deduplicated versions evaluate 13,937 samples zero shot and 14,015 five shot, after removing exact repeats. With log probability scoring, the harness scores all four answer letters for every question.

What is a good MMLU score?

It depends on the model's size and the settings. Random guessing gives 25 percent, and the original authors estimated expert human accuracy at about 89.8 percent. For your own work, the more useful question is whether a change moved the score beyond its standard error under identical settings.

Should I use MMLU or MMLU-Redux?

Run full MMLU for comparability with older results, and add MMLU-Redux when you want a cleaner signal. Redux keeps the original questions but labels which ones its annotators found to be wrong or unclear, and the harness ships a generative task for it.

Can I run only some MMLU subjects?

Yes. In lm-evaluation-harness, name a group such as mmlu_stem or a single subject such as mmlu_anatomy in --tasks. In Inspect, pass -T subjects=anatomy,astronomy. Small subjects have large error bars, so treat single subject results as hints, not verdicts.