github.com · Benchmarks
Massive Multitask Language Understanding: a multiple choice benchmark spanning 57 subjects from math to law, introduced in 2020 and now largely saturated by frontier models.
MMLU is one of 13 benchmark tools on TrueIQ. See all benchmark tools or how we check facts.
// guides
How to use MMLU benchmark runs: pick the right variant, run it with lm-evaluation-harness or Inspect, smoke test the format, and read the score honestly.
MMLU vs GPQA explained: broad exam recall versus expert written science questions, why GPQA scores run lower, and which benchmark still means anything in 2026.
The GDPval benchmark tests AI on real work from 44 occupations. How tasks and grading work, how to read the GDPval-AA leaderboard, and what it misses.
MMLU vs SWE-bench explained: what each measures, why the scores are not comparable, and which benchmark to trust for knowledge vs coding agents in 2026.
MMLU vs MMLU Pro explained: how the 4 option and 10 option benchmarks differ, why their scores aren't comparable, and which one to trust for LLMs in 2026.
SWE-Bench Pro explained: how the Scale AI coding benchmark differs from Verified, what changed in V2, and how to read public and private leaderboard scores.
// related
MMLU-Progithub.comHarder successor to MMLU from TIGER-Lab: about 12,000 reasoning heavy questions in 14 disciplines with ten answer options, published at NeurIPS 2024.BenchmarksFree★ 433
SWE-benchswebench.comOriginal Princeton SWE-bench of real GitHub issues where a fix counts only if the repo's tests pass; includes Lite, Verified and related subsets. For Scale AI's harder Pro benchmark, see SWE-Bench Pro.BenchmarksFree★ 6.1k
Humanity's Last Exam (HLE)lastexam.ai2,500 expert written questions from CAIS and Scale AI, plus HLE Diamond, a cleaned 1,000 question subset (500 reasoning, 500 knowledge) released September 2026.BenchmarksFree★ 1.7k
ARC-AGIarcprize.orgAbstraction and Reasoning Corpus series from Francois Chollet and the ARC Prize Foundation that tests skill acquisition on novel tasks; ARC-AGI-3 adds interactive games.BenchmarksFree★ 4.9kBIG-benchgithub.comBeyond the Imitation Game: a collaborative benchmark of 200+ diverse tasks from 450 authors at 132 institutions, built to probe and extrapolate language model capabilities.BenchmarksFree★ 3.2k
SWE-Bench Prolabs.scale.comScale AI coding agent benchmark of long horizon GitHub style tasks with public, private and held out splits; V2 refreshes the public set to 642 validated tasks plus a HARD-51 subset.BenchmarksFree★ 555