// category · 13 tools · 0 open source
LLM Benchmarks
The standard tests used to compare models on knowledge, reasoning, coding and abstraction. Check what each one measures, how it is scored and whether results may be contaminated.
Benchmarks are fixed test sets that let you compare AI models on the same questions under the same rules. This category lists the benchmarks that appear most often in model launch posts and leaderboards, from older knowledge exams to recent agent and coding tests, with a short note on what each one measures.
A benchmark is only useful while it still separates strong models from weak ones. Older exams such as MMLU saturated once top models passed 90 percent, which is why harder sets followed: MMLU Pro, GPQA for graduate level science, and Humanity's Last Exam, a set of 2,500 expert written questions from the Center for AI Safety and Scale AI. In September 2026 the HLE authors also released HLE Diamond, a cleaned 1,000 question subset split into reasoning and knowledge halves; our HLE Diamond guide explains what it is and how to read its scores.
Other entries test different skills. SWE-bench and SWE-Bench Pro measure whether a model can fix real software issues. ARC-AGI tests learning a new rule from a few examples rather than recalling knowledge. GDPval grades work products from real occupations. RAID is different again: it benchmarks AI text detectors, not language models.
Before you trust a reported score, check five things:
- Which version of the set. Full, text only, verified or a cleaned subset can all carry the same name.
- Tools or no tools. Web search and code execution can add more than 20 points.
- Settings. Reasoning effort, token limits and the judge model all move results.
- Who ran it. A lab's own number and an independent run often differ.
- Noise. A one or two point gap on a few hundred questions may mean nothing.
Contamination is the other risk. When test questions leak into training data, a benchmark starts measuring memory instead of skill, which is why many datasets are gated and carry canary strings.
For a full walkthrough, read LLM benchmarks explained, MMLU vs MMLU Pro and MMLU vs SWE-bench. To see how benchmark scores are combined into rankings, browse the leaderboards category.
SWE-benchswebench.comOriginal Princeton SWE-bench of real GitHub issues where a fix counts only if the repo's tests pass; includes Lite, Verified and related subsets. For Scale AI's harder Pro benchmark, see SWE-Bench Pro.BenchmarksFree★ 6.1k
Humanity's Last Exam (HLE)lastexam.ai2,500 expert written questions from CAIS and Scale AI, plus HLE Diamond, a cleaned 1,000 question subset (500 reasoning, 500 knowledge) released September 2026.BenchmarksFree★ 1.7k
ARC-AGIarcprize.orgAbstraction and Reasoning Corpus series from Francois Chollet and the ARC Prize Foundation that tests skill acquisition on novel tasks; ARC-AGI-3 adds interactive games.BenchmarksFree★ 4.9k
BIG-benchgithub.comBeyond the Imitation Game: a collaborative benchmark of 200+ diverse tasks from 450 authors at 132 institutions, built to probe and extrapolate language model capabilities.BenchmarksFree★ 3.2kMMLUgithub.comMassive Multitask Language Understanding: a multiple choice benchmark spanning 57 subjects from math to law, introduced in 2020 and now largely saturated by frontier models.BenchmarksFree★ 1.6k
SWE-Bench Prolabs.scale.comScale AI coding agent benchmark of long horizon GitHub style tasks with public, private and held out splits; V2 refreshes the public set to 642 validated tasks plus a HARD-51 subset.BenchmarksFree★ 555GPQAgithub.comGraduate level, Google proof science Q&A: 448 expert written biology, physics and chemistry questions, with a 198 question Diamond subset widely reported in model launches.BenchmarksFree★ 545MMLU-Progithub.comHarder successor to MMLU from TIGER-Lab: about 12,000 reasoning heavy questions in 14 disciplines with ten answer options, published at NeurIPS 2024.BenchmarksFree★ 433
Infinity-Chathuggingface.coOpen dataset from the Artificial Hivemind paper: 26,070 real open ended chatbot questions plus 31,250 human ratings, used to measure how alike model answers are.BenchmarksFree★ 77
NoveltyBenchnovelty-bench.github.ioCarnegie Mellon benchmark for output diversity: 1,100 prompts, each answered 10 times, scored on how many answers are meaningfully different and how useful the new ones are.BenchmarksFree★ 35
FrontierCodecognition.comCognition's private coding agent benchmark that grades whether maintainers of real open source repositories would merge an agent's change, scoring scope, tests and code quality as well as correctness.BenchmarksSee site
GDPvalopenai.comOpenAI benchmark of real professional work: 1,320 expert written tasks across 44 occupations, with a public gold set of 220 tasks graded against human expert deliverables.BenchmarksFree
RAIDraid-bench.xyzOpen benchmark and leaderboard for AI text detectors from the University of Pennsylvania: over 6 million texts across 11 generators, 8 domains and 11 attacks, scored at a fixed false positive rate.BenchmarksFree// guides
Guides to LLM benchmarks
How to Use the MMLU Benchmark, Step by Step
How to use MMLU benchmark runs: pick the right variant, run it with lm-evaluation-harness or Inspect, smoke test the format, and read the score honestly.
MMLU vs SWE-bench: Which Should You Use?
MMLU vs SWE-bench explained: what each measures, why the scores are not comparable, and which benchmark to trust for knowledge vs coding agents in 2026.
LLM Benchmarks Explained: What the Scores Mean
LLM benchmarks explained: what MMLU, GPQA Diamond, Humanity's Last Exam, SWE-bench and ARC-AGI measure, why scores saturate, and how to spot contamination.
MMLU vs GPQA: Key Differences and Which to Use
MMLU vs GPQA explained: broad exam recall versus expert written science questions, why GPQA scores run lower, and which benchmark still means anything in 2026.
GDPval Benchmark Explained: Tasks, Scores, Limits
The GDPval benchmark tests AI on real work from 44 occupations. How tasks and grading work, how to read the GDPval-AA leaderboard, and what it misses.
MMLU vs MMLU Pro: Key Differences and Which to Use
MMLU vs MMLU Pro explained: how the 4 option and 10 option benchmarks differ, why their scores aren't comparable, and which one to trust for LLMs in 2026.
SWE-Bench Pro Explained: V1, V2, Scores
SWE-Bench Pro explained: how the Scale AI coding benchmark differs from Verified, what changed in V2, and how to read public and private leaderboard scores.
FrontierCode Benchmark Explained: How It Scores
The FrontierCode benchmark explained: how Cognition grades whether maintainers would merge AI code, what Main and Diamond mean, and how to read the leaderboards.
HLE Diamond Leaderboard and Scores (October 2026)
HLE Diamond leaderboard and scores for October 2026: the 1,000 question Humanity's Last Exam subset, Scale and lastexam.ai results, with and without tools.
LLM Leaderboards Compared: Which Ones to Trust
Every major LLM leaderboard compared: how Arena, Artificial Analysis, LiveBench, SEAL, Epoch AI and Vellum score models, and where each one can mislead you.
Coding Agent Benchmark Rankings (October 2026)
Coding agent benchmark rankings from SWE-Bench Pro, FrontierCode, Terminal-Bench and the Artificial Analysis index, with sourced scores, dates and why they differ.
Artificial Hivemind: Why AI Models Sound Alike
The Artificial Hivemind paper explained: why AI models give the same answers, what Infinity-Chat and NoveltyBench measure, and how to test output diversity yourself.
LiveBench vs Artificial Analysis: Key Differences
LiveBench vs Artificial Analysis: how the two AI leaderboards differ on tasks, grading, weighting and cost, why their rankings disagree, and when to trust each.
What Is Claudish? Why AI Text Is Hard to Read
What is Claudish? The dense style of AI coding agents, what a new study found about why AI text is hard to read, and what it means for spotting AI writing.
How to Detect AI Writing: A Step by Step Guide
How to detect AI writing: read for the patterns, check facts and sources, run two detectors, ask for drafts, then talk to the writer. A careful method, free.
LLM Confidence Scores: Can You Trust Them?
Can you trust an LLM confidence score? What verbalized scores, logprobs and sampling measure, what 2026 calibration research shows, and how to set safe thresholds.
RAID Benchmark: How to Read the AI Detector Board
The RAID benchmark tests AI text detectors on 6 million samples with attacks. What it measures, how to read the leaderboard, and why 99 percent scores mislead.