LiveBench vs Artificial Analysis: Key Differences

LiveBench vs Artificial Analysis comes down to one question: do you want a single scored exam or a full buying guide? LiveBench is a benchmark. Since its 2.0 relaunch in October 2026 it gives models long, real work in seven suites and grades what they hand back, mostly with scripts and hidden tests. Artificial Analysis is an independent measurement service. It combines ten evaluations into its Intelligence Index and also measures price, speed and latency for more than 250 models. Use LiveBench to see who finishes hard agent tasks, and Artificial Analysis to pick a model you can afford to run.

Both boards changed a lot this year, and many comparisons online still describe LiveBench as a short quiz of math and puzzle questions. That version (v1) still exists in the release archive, but the current board is a different test. This guide compares the versions live on October 27, 2026, using each project's own methodology pages and published data.

LiveBench vs Artificial Analysis at a glance

The short version: LiveBench answers "can this model do the job?" while Artificial Analysis answers "how capable is it, and what will it cost me per task and per second?"

What LiveBench measures in 2026

LiveBench started in 2024 as a contamination limited benchmark. The original paper (an ICLR 2025 spotlight) set three rules: questions drawn from recent sources, answers scored against objective ground truth without an LLM judge, and a broad spread of tasks. That version had 18 tasks across math, coding, reasoning, language, instruction following and data analysis. Each update replaced about one sixth of the questions, and new questions stayed private for a month before release (paper).

LiveBench 2.0 arrived with the release dated October 14, 2026, and a second 2.0 release followed on October 25. The leaderboard now describes itself as a long horizon benchmark where models produce decks, research reports, bug fixes and apps that run for hours. The Oct 25 release data lists seven suites:

• Long Agentic (18 tasks): take over a working business app (payroll, budgets, ticketing, stock records) and keep it correct through rounds of changing rules, or run a codebase through simulated months of releases. Graded by scripts against hidden reference data.
• Coding (29 tasks): fix a real bug that a merged pull request once fixed, in a frozen production codebase. Hidden tests check the fix, and a fix that breaks existing behavior scores 0.
• Browser Use (25 tasks): multistep jobs in working web apps (a shop, a helpdesk, a wiki) through a real browser that the model sees by screenshot. A model that cannot read screenshots scores 0.
• Research (22 tasks): answer an open brief from a named person, with sources. Graded against a task specific rubric.
• Design (25 tasks): produce or rework Word documents, PDFs and slide decks, scored on visual and structural checks of the rendered file.
• Data Analysis (150 questions) and Reasoning (202 questions): short questions with one correct answer, no tools.

Five of the seven suites are agentic. Each cell is a mean task score from a single run, and a task the harness could not measure even after one rerun counts as 0. Models run at their highest supported effort setting.

How the LiveBench overall score is weighted

The Oct 25 release weights Long Agentic 2 times, Design and Coding 1.5 times each, and the other suites once. The Oct 14 release weighted every suite equally, and LiveBench says a published release never changes. So an overall score from Oct 14 and one from Oct 25 are not directly comparable, even for the same model.

What Artificial Analysis measures

Artificial Analysis runs every model itself under identical prompts and settings. Its Intelligence Index v4.3.2 combines ten evaluations in four weighted groups (methodology):

Four of the ten are marked private: AA-Briefcase, AutomationBench-AA, AA-Omniscience and CritPt. The index is English and text focused; multilingual, vision and other tests are reported separately and do not count toward it. Reasoning models run at temperature 0.6 and other models at 0, and scores are pass@1 averaged over repeats where an evaluation has them.

Grading varies by test. Terminal-Bench and SciCode use test suites. HLE and AA-LCR use an LLM equality checker. GDPval-AA and AA-Briefcase use pairwise comparisons by a panel of three LLM judges from different labs, turned into Elo ratings. We explain that Elo setup in our GDPval benchmark guide, and HLE in our HLE Diamond guide.

The second half of the site is performance data. Artificial Analysis tests each API endpoint eight times a day for its 1,000 and 10,000 token workloads and reports the median over the past 72 hours for output speed and time to first token (performance methodology). The default speed shown on the site uses the 10,000 token workload. That is data LiveBench does not try to collect.

The real differences that change rankings

Pass or fail work versus a weighted capability mix

LiveBench rewards finishing long jobs. Its heaviest suite asks a model to keep an app correct while requirements change, which punishes early shortcuts. The Intelligence Index spreads weight across knowledge (AA-Omniscience also penalizes hallucinated answers), science problems, long documents and agent work. A model can score well on facts and science and still lose ground on LiveBench's long app tasks, and the reverse.

One effort setting versus every effort setting

LiveBench lists one row per model at maximum effort. Artificial Analysis lists separate rows for max, xhigh, high, medium and low effort where a lab offers them, so Claude Opus 5.5 appears five times. That makes Artificial Analysis better for asking "how much do I lose at medium effort?" and LiveBench simpler to scan.

Grading style

Most LiveBench 2.0 suites are graded by scripts, hidden tests or ground truth, which follows the v1 argument that LLM judges make large errors on hard questions. Research is the exception: it uses a rubric. Artificial Analysis uses LLM judges where the task has no single right answer, but it uses panels from three labs to limit self preference. If you distrust model graders, our LLM as a judge guide explains where they fail.

Single run versus repeats

LiveBench reports a single run, and a harness fault on a rerun still counts as zero. Artificial Analysis repeats several evaluations up to five times. With only 18 to 29 tasks in several LiveBench suites, one task is worth several points of a suite score, so small gaps are noise. See how to evaluate LLM model performance for why sample size matters.

Cost metrics that mean different things

LiveBench shows cost per successful task: total cost divided by tasks, then divided by the score, so failures and partial credit raise it. Artificial Analysis shows cost per task to run its index tasks, based on the token counts each provider reports. Neither number is a quote for your workload, and the two should never be compared with each other.

Where the two boards agree and disagree

On October 27, 2026, the top of each board looked like this (figures as published on each site that day):

The same Anthropic and OpenAI families sit at the top of both boards, but the order differs. Claude Fable 5.1 leads LiveBench while Claude Opus 5.5 leads Artificial Analysis. Kimi K3.1 Pro shows the biggest gap: fifth row on Artificial Analysis, tenth on LiveBench, where its Long Agentic score is 39.1 against 77.9 for Fable 5.1. The likely reason is weighting. LiveBench doubles the long agentic suite, and Artificial Analysis gives knowledge and science tasks half the index. That is our reading of the published weights, not a claim either project makes.

Cost tells another story. On LiveBench, GPT-6.1 Sol scores 84.1 at $0.761 per successful task while Claude Fable 5.1 scores 85.7 at $6.904. A 1.6 point gap for about nine times the cost is the kind of tradeoff both sites now chart.

Which one should you use?

• Choosing a model for an agent product: start with LiveBench's Long Agentic, Coding and Browser Use columns, then check the same models on Artificial Analysis for speed and cost.
• Comparing effort levels or providers: use Artificial Analysis. It is the only one of the two with per effort rows, latency and per provider speed.
• Tracking open weights models: both mark them. Artificial Analysis covers far more (65 of 132 ranked models are open weights).
• Checking for contamination: LiveBench's design goal is fresh tasks, and v1 held questions back for a month. The 2.0 pages we read do not state a refresh schedule, so treat that as unconfirmed for now.
• Writing a report or citing a number: cite the release date for LiveBench and the index version (v4.3.2) for Artificial Analysis, because both change their formulas.

Whichever board you pick, confirm on your own tasks before you commit. Our LLM leaderboards guide covers Arena, Epoch AI and SEAL as third opinions, and coding agent benchmark rankings goes deeper on coding boards such as SWE-bench.

A note on who runs each board: LiveBench is sponsored by Abacus.AI, its site says the results power an Abacus AI subscription product, and Abacus models such as Smaug-Nemotron-3.5-NVFP4 appear on the board. Artificial Analysis publishes rules that bar providers from serving its test traffic differently from ordinary traffic. Neither fact makes a board wrong, but both tell you who is behind the numbers.

FAQ

Is LiveBench better than Artificial Analysis?

Neither is better overall. LiveBench is the stronger test of long agent work graded by scripts. Artificial Analysis is broader, covers many more models and adds price and speed data. Most teams should read both.

What is LiveBench 2.0?

LiveBench 2.0 is the relaunch of LiveBench released on October 14, 2026. It replaced the short question set of v1 with seven suites, five of them agentic, covering long running apps, real bug fixes, browser tasks, research reports and document design. Older v1 releases remain viewable in the release picker.

What is the Artificial Analysis Intelligence Index?

It is a weighted average of ten evaluations run by Artificial Analysis under identical settings. Version 4.3.2 weights agents 30%, coding 20%, scientific reasoning 20% and general tasks 30%. Our LLM benchmarks explained guide covers several of the component tests.

Why do LiveBench and Artificial Analysis rank models differently?

They test different work and weight it differently. LiveBench doubles its long agentic suite and runs each model once at maximum effort. Artificial Analysis spreads weight across knowledge, science, long documents and agents, and lists each effort level separately. Small gaps on either board are also within normal run to run noise.

Does LiveBench use an LLM judge?

Mostly no. Six of its seven suites are graded by scripts, hidden tests, ground truth or rendered file checks. The Research suite is graded against a task specific rubric. The original v1 paper argued against LLM judges because they made large errors on hard math and puzzle questions.

Is LiveBench still contamination free?

LiveBench still describes itself that way. In v1 that claim rested on regular question refreshes and a one month delay before new questions went public. The 2.0 material we reviewed does not describe a refresh schedule, so check its release notes before relying on that claim.