MMLU vs SWE-bench: Which Should You Use?

MMLU vs SWE-bench is not a close call between two versions of the same test. MMLU is a multiple choice knowledge exam across 57 academic subjects; SWE-bench is a coding agent benchmark where a model must edit a real GitHub repository so the project's tests pass. Use MMLU (or better, MMLU-Pro) when you care about broad pretrained knowledge. Use the SWE-bench family when you care about whether an agent can fix real software issues. Never rank models by mixing an MMLU percentage with a SWE-bench resolve rate. Preparing for a human coding assessment on HackerRank or Codility? See how to take a coding test.

This guide puts the two side by side, explains what each number actually means in 2026, and gives a simple pick guide so you stop treating launch slide tables as if every column measured the same skill. For the wider map of knowledge, coding and agentic suites, start with LLM benchmarks explained.

MMLU vs SWE-bench at a glance

In plain words: MMLU asks "does the model know this?" SWE-bench asks "can the model change this codebase until the tests go green?"

What MMLU measures

MMLU, short for Massive Multitask Language Understanding, was introduced by Dan Hendrycks and colleagues to test how much knowledge a model picks up during pretraining (paper). Its questions span 57 subjects, from elementary mathematics to professional law and medicine, collected from freely available practice exams and course material. The authors estimated expert level accuracy at roughly 89.8 percent. When the paper came out, the largest GPT-3 model scored 43.9 percent in the few shot setting.

For years MMLU was the default headline number in model cards. That popularity is also why it stopped separating frontier systems: scores bunched near the ceiling, and leftover gaps turned out to be partly answer key noise rather than missing knowledge. Follow ons such as MMLU-Pro (ten options, harder items) and cleaned subsets such as MMLU-Redux exist because of those problems. Our MMLU vs MMLU Pro guide covers that lineage in detail.

What MMLU cannot tell you is whether a model can write, edit or ship software. Every item is a selection among given options. Generation quality, tool use, repository navigation and "I do not know" behavior are out of scope.

What SWE-bench measures

SWE-bench asks a different question. The original paper builds 2,294 task instances from real GitHub issues and merged pull requests across twelve popular Python repositories (paper; leaderboard). The model (usually wrapped in an agent harness) sees the issue text and the repository at the parent commit. Success means producing a patch that makes the repository's fail to pass tests succeed without breaking the pass to pass regression suite. The model does not see those tests while it works.

That design is closer to real engineering work than HumanEval style function completion. Gold patches in the original set often touch more than one file, and codebases average hundreds of thousands of lines. Early published baselines were low: with BM25 retrieval, Claude 2 resolved only about 2 percent of issues. Later agent scaffolds pushed resolve rates much higher, which is why a SWE-bench number is almost always a score for a (model, harness, budget) triple, not for a naked model.

The family grew splits that people still mix up:

• SWE-bench (full): 2,294 instances.
• Lite: 300 cheaper instances for faster iteration.
• Verified: 500 human filtered instances that became the default coding score for a while.
• Multilingual / Multimodal: later boards on swebench.com for other languages and visual issues.
• SWE-Bench Pro: Scale AI's longer horizon follow on with public and private splits; OpenAI's recommended reporting target after February 2026.

Why the scores are not comparable

A 90 percent MMLU score and a 70 percent SWE-bench resolve rate do not mean "knowledge beats coding" or the reverse. Three structural differences make cross benchmark ranking meaningless.

Different skills. MMLU rewards recall and short reasoning inside a closed answer set. SWE-bench rewards finding the right files, writing a patch that fits the project's style, and surviving hidden tests. A model can ace academic quizzes and still fail to resolve a Django issue.

Different ceilings and failure modes. MMLU fails when the answer key is wrong, the question leaked into training data, or the model guesses among four options. SWE-bench fails when the harness is weak, the tests are too narrow or too wide, the issue is underspecified, or the gold fix appeared in training data. In February 2026 OpenAI stopped reporting SWE-bench Verified for frontier launches after auditing hard tasks: at least 59.4 percent of the audited problems had material test or description flaws, and frontier models could reproduce gold patches for some tasks (OpenAI). The same post recommended reporting the public split of SWE-Bench Pro instead.

Different units. MMLU accuracy is "correct letters over questions." SWE-bench resolve rate is "instances whose tests all pass after one agent attempt under a stated setup." Changing the harness, turn budget or network policy can move SWE-bench more than swapping the base model. Changing the prompt style or few shot count can move MMLU by several points even when the model is fixed.

Which should you use?

Pick by the decision you are trying to make.

• Shortlisting a general purpose chat or reasoning model: prefer harder knowledge and reasoning exams than classic MMLU, such as GPQA Diamond or Humanity's Last Exam, and treat MMLU as a historical baseline. See LLM leaderboards compared for which aggregators still include it.
• Evaluating coding agents or developer copilots that edit repos: use the SWE-bench family, preferably Pro (with version and harness named), and always run a small internal suite of your own tickets.
• Tracking a fine tune's knowledge retention: MMLU or MMLU-Pro as a smoke test is fine, as long as you keep the exact harness fixed before and after.
• Reading a launch slide that stacks both columns: treat them as two different product claims. Ask which SWE-bench split, which harness, and whether Verified or Pro was used.

If you only have room for one public coding number in 2026, Pro is the safer citation than Verified for frontier claims. If you only have room for one knowledge number, MMLU-Pro beats classic MMLU for mid tier models, and neither should be your main signal at the absolute frontier.

How to run or read each one

MMLU / MMLU-Pro. EleutherAI's lm-evaluation-harness ships task groups such as mmlu and mmlu_pro. Report the exact task name, shot count and whether chain of thought was used. For older papers that only list "MMLU," assume the four option original unless they say otherwise. Our MMLU step by step guide covers variants, flags and smoke tests.

SWE-bench. Use the official evaluation harness and the split named on swebench.com. Record agent framework, max turns or tokens, and whether the run was vendor reported or independently checked. Do not compare a Lite score to Verified or Pro. For Pro versions and private vs public splits, see our SWE-Bench Pro guide.

For building an eval around your own product rather than quoting a board, follow how to evaluate LLM applications.

FAQ

Is MMLU or SWE-bench better?

Neither is universally better. MMLU is better for broad knowledge comparisons; SWE-bench is better for coding agent comparisons. Choose the benchmark that matches the skill you need.

Can I compare MMLU and SWE-bench scores on the same chart?

You can plot them as separate axes, but you should not average them or rank models by mixing the percentages. They measure different abilities on different scales.

What replaced SWE-bench Verified?

OpenAI stopped reporting Verified in February 2026 and recommended the public split of SWE-Bench Pro. Verified still appears in older papers and some third party boards, so always check the date and the exact split name.

Is MMLU still useful in 2026?

Yes, as a saturated baseline and for comparing smaller or open weight models where scores still spread out. For frontier launches it is weak differentiation; prefer harder exams and task specific evals.

Does SWE-bench test only Python?

The original SWE-bench and Verified sets are Python centric. Later boards on swebench.com add multilingual and multimodal variants. Always name the board you are citing.

Why do two labs report different SWE-bench numbers for the same model?

Because the resolve rate depends on the agent scaffold, tool set, budget and sometimes the split. A high score with a strong harness is not the same claim as a score with a minimal bash only agent.