github.com · Benchmarks
Beyond the Imitation Game: a collaborative benchmark of 200+ diverse tasks from 450 authors at 132 institutions, built to probe and extrapolate language model capabilities.
BIG-bench is one of 13 benchmark tools on TrueIQ. See all benchmark tools or how we check facts.
// guides
How to use MMLU benchmark runs: pick the right variant, run it with lm-evaluation-harness or Inspect, smoke test the format, and read the score honestly.
LLM benchmarks explained: what MMLU, GPQA Diamond, Humanity's Last Exam, SWE-bench and ARC-AGI measure, why scores saturate, and how to spot contamination.
// related
Humanity's Last Exam (HLE)lastexam.ai2,500 expert written questions from CAIS and Scale AI, plus HLE Diamond, a cleaned 1,000 question subset (500 reasoning, 500 knowledge) released September 2026.BenchmarksFree★ 1.7k
ARC-AGIarcprize.orgAbstraction and Reasoning Corpus series from Francois Chollet and the ARC Prize Foundation that tests skill acquisition on novel tasks; ARC-AGI-3 adds interactive games.BenchmarksFree★ 4.9kGPQAgithub.comGraduate level, Google proof science Q&A: 448 expert written biology, physics and chemistry questions, with a 198 question Diamond subset widely reported in model launches.BenchmarksFree★ 545MMLU-Progithub.comHarder successor to MMLU from TIGER-Lab: about 12,000 reasoning heavy questions in 14 disciplines with ten answer options, published at NeurIPS 2024.BenchmarksFree★ 433
SWE-benchswebench.comOriginal Princeton SWE-bench of real GitHub issues where a fix counts only if the repo's tests pass; includes Lite, Verified and related subsets. For Scale AI's harder Pro benchmark, see SWE-Bench Pro.BenchmarksFree★ 6.1kMMLUgithub.comMassive Multitask Language Understanding: a multiple choice benchmark spanning 57 subjects from math to law, introduced in 2020 and now largely saturated by frontier models.BenchmarksFree★ 1.6k