FrontierCode Benchmark Explained: How It Scores

The FrontierCode benchmark is Cognition's test of whether an AI coding agent writes changes that a real open source maintainer would merge. Passing hidden tests is only the first hurdle: each attempt is also graded on scope, test quality, style and the conventions of the repository, and any failed "blocker" check scores zero. The tasks are private, version 1.1 is current, and scores are reported on two subsets, Main (100 tasks) and Extended (150 tasks).

It became much more visible this week. On October 23, 2026, Artificial Analysis launched version 2.0 of its Coding Agent Index and made FrontierCode v1.1 one of its three components, replacing DeepSWE. Searches for the benchmark now autocomplete to "frontiercode 1.1", "frontiercode leaderboard", "frontiercode diamond" and "frontiercode 1.1 main", which are exactly the terms that confuse people. This guide explains what each one means and how far to trust the numbers.

What the FrontierCode benchmark measures

Most coding benchmarks ask one question: does the patch make the hidden tests pass? FrontierCode asks a harder one: would the maintainer approve this pull request? Cognition introduced it on June 8, 2026 with that framing, and it grades six things:

• Behavioral correctness: does the change solve the problem?
• Regression safety: does it break anything else?
• Mechanical cleanliness: does it pass the project's build, lint and format checks?
• Test correctness: do the tests the agent wrote actually capture the behavior?
• Scope: does it touch only what it needs to?
• Code quality: does it follow the codebase's conventions and stay readable?

The motivation is a known gap in older benchmarks. In March 2026, METR had four maintainers of scikit-learn, Sphinx and pytest review 296 AI written pull requests that had passed the SWE-bench Verified grader. Their write up found that roughly half of those test passing patches would not have been merged, and that the automated grader ran about 24 percentage points above maintainer decisions on average. Even the original human patches were only accepted 68% of the time in that blind review, which shows how subjective the last step of code review is. FrontierCode tries to put that judgment into the grader itself.

How FrontierCode tasks are built

Cognition worked with the maintainers of 36 open source repositories. More than 20 developers wrote the tasks, spent over 40 hours on each one, and defined what "mergeable" means in their own project. Every task was then reviewed by a Cognition researcher.

A few design choices set it apart from SWE-Bench Pro and its relatives:

• Short, human prompts. Each prompt has a brief task description plus the repository's testing, lint and style guidelines, similar to an AGENTS.md file. Cognition says its task descriptions are about a third the length of SWE-Bench Pro's, so the agent has to infer intent the way a new contributor would.
• Hand picked tasks. Maintainers chose tasks from chains of related pull requests and freeform requests rather than scraping single issues, and Cognition says the set covers three times as many programming languages as SWE-Bench Pro.
• Difficulty from quality, not size. Patches are smaller than in some rival benchmarks. The difficulty comes from the rubric.
• Adversarial review. Each task author tried to pass with a lazy or deliberately wrong solution (to catch false passes), wrote a valid alternative solution (to catch false failures), and wrote four solutions aimed at different scores to check that the rubric separates good from mediocre work.

Cognition reports 81% fewer misclassification errors than SWE-Bench Pro. That figure comes from Cognition's own trajectory analysis, so treat it as a vendor claim until someone reproduces it.

How FrontierCode scores an attempt

The grader mixes deterministic checks with an LLM review. Cognition lists six check types:

Each criterion is either a blocker or a nonblocker. Blockers are things a maintainer would refuse to merge over, such as a wrong result, a broken build or edits to unrelated files. Nonblockers are quality signals such as type safety or readability.

Two numbers come out of this:

1. Pass rate: the share of attempts that clear every blocker.
2. Score: zero if any blocker fails. If all blockers pass, a weighted total of the rubric. Artificial Analysis describes the result as landing between 70% and 100% for a passing attempt, higher when it also meets the nonblockers.

So a score of 50% does not mean half the code was good. It means a mix of zeros (failed attempts) and passing attempts in the 70% to 100% band, averaged across tasks.

FrontierCode 1.1, Main, Extended and Diamond

FrontierCode launched with three nested subsets: Extended (all 150 tasks), Main (the 100 hardest) and Diamond (the 50 hardest). On July 7, 2026, Cognition released version 1.1 with three changes:

• Fair internet use. Agents may still search the web and read documentation, because some tasks need it. A prompt now spells out what is allowed, and a scanner sets any run that opens the upstream pull request, a mirror or another solution source to zero. Cognition says that with the prompt in place, unfair use fell below 1% for every model it tested. It tried a blocklist first, but the list grew to about 1,200 domains and agents kept finding ways around it, sometimes spending more than 20 turns fighting it.
• Fairer grading. Cognition audited more than 1,000 criteria and demoted 75 overly strict blockers to nonblockers, to cut false failures.
• Diamond retired. After the changes, Diamond no longer held the 50 hardest tasks, and solve rates on the hardest tasks were so low that the subset was noisy. Scores are now reported on Main and Extended only.

That is why searches for "frontiercode diamond" turn up old numbers. In the June launch post, the best Diamond score was 13.4% (Claude Opus 4.8), with GPT-5.5 at 6.3% and Gemini 3.1 Pro at 4.7%. Those were version 1.0 results, and Diamond is no longer reported, so don't compare them with anything current.

Two FrontierCode leaderboards, two harnesses

There are now two places to read FrontierCode numbers, and they measure slightly different things.

Cognition's own leaderboard at cognition.com/frontiercode reports the 1.1 methodology. In its launch post, Cognition ran each model five times at every reasoning effort and reported the best effort.

Artificial Analysis started running the benchmark independently on October 12, 2026. It runs the 100 Main tasks three times each, puts every model in its open source Stirrup harness with a 250 turn cap, and uses GPT-6 Luna (medium) as the LLM judge. Its launch results:

• Claude Opus 5.5 (max with fallback) led with a 52% score and a 58% pass rate, so leading models still failed more than 40% of attempts.
• GPT-6.1 Sol (max) and Claude Fable 5.1 (max with fallback) followed at 49%.
• Nine models from six labs scored between 43% and 52%.
• 9 of the 100 tasks were passed by no model on any attempt.
• Cost per task ranged from $0.09 to $12.00.

Then on October 23, the Artificial Analysis Coding Agent Index v2.0 put FrontierCode inside a composite. The index scores a model together with its agent harness across 290 tasks, weighting three benchmarks equally: FrontierCode v1.1 (100 tasks), Terminal-Bench 4.0 (66 terminal tasks) and SWE-Atlas-QnA-AA (Scale AI's 124 repository question tasks, with 127 of 1,306 rubric criteria reworded and GPT-6 Luna at xhigh as judge, five votes per criterion). Claude Code with Claude Opus 5.5 (max) led the index at 61.7, ahead of Claude Code with Claude Sonnet 5.5 (max) at 59.3.

Look at the FrontierCode part of that index and you see a lesson in harnesses. Opus 5.5 scored 56.0% on FrontierCode inside Claude Code, against 52% in the Stirrup harness on October 12. Different harness, different date, different number. When you see a FrontierCode score, check which harness and which effort setting produced it before you compare.

How to read a FrontierCode score

Use this checklist before you quote or act on a number:

1. Which version? Only 1.1 numbers are comparable with today's leaderboards. Anything labeled Diamond is from 1.0.
2. Which subset? Main (100 tasks) and Extended (150) are different difficulty mixes. Artificial Analysis reports Main.
3. Score or pass rate? Pass rate tells you how often the agent clears every hard requirement. Score adds polish on top.
4. Which harness and effort? Stirrup, Claude Code and Codex results differ for the same model, and "max" settings cost far more.
5. How close is the gap? The middle of the board is packed. In the index, 15 of 24 configurations sat between 43% and 49% on FrontierCode, so a two point gap may not survive a rerun. Artificial Analysis averages only three attempts per task.
6. What does it cost? Both leaderboards publish cost per task. These are pay per token API costs, not subscription prices.

Limits of the FrontierCode benchmark

FrontierCode is a strong step toward realistic grading, but it has real limits:

• You can't inspect the tasks. The test set is private to avoid contamination, so outsiders can't audit the rubrics or rerun the exact grading. You are trusting Cognition and Artificial Analysis.
• An LLM sits in the grader. Prompt review and adaptive tests depend on a judge model, with the usual risks covered in our guide to LLM as a judge.
• The builder also sells an agent. Cognition makes the Devin coding agent. Opening the benchmark to outside evaluators such as Artificial Analysis helps, but keep the incentive in mind.
• One shot, no feedback. Like METR's study, the benchmark grades a single submission. Real contributors revise after review.
• Small sample. With 100 Main tasks, each task is one point of the average, so small differences are noise.

To see how FrontierCode ranks agents next to SWE-Bench Pro and Terminal-Bench, see our coding agent benchmark rankings. For a wider view of how coding scores relate to other evals, see MMLU vs SWE-bench, our LLM benchmarks explainer and the leaderboard comparison. If you're choosing a model for your own codebase, the method in how to evaluate LLM model performance matters more than any public number: borrow FrontierCode's idea and grade candidate patches against your own team's review rules. Independent runs on the Epoch AI Benchmarking Hub are another useful cross check for older coding benchmarks.

FAQ

What is the FrontierCode benchmark?

A private coding agent benchmark from Cognition, launched in June 2026, that grades whether a change to a real open source repository would be merged by its maintainer. It checks correctness, regressions, tests, scope, style and code quality, and any failed blocker scores zero.

What is FrontierCode 1.1?

The current version, released July 7, 2026. It added fair internet use rules with a scanner that zeroes runs that look up the solution, demoted 75 overly strict blockers, and retired the Diamond subset.

What is FrontierCode Diamond?

The 50 hardest tasks in the original release. Cognition stopped reporting it in version 1.1 because it no longer matched the hardest tasks and its scores were too noisy. Old Diamond numbers, such as the 13.4% top score in June, are not comparable with current results.

What is the difference between FrontierCode Main and Extended?

Main is 100 tasks and Extended is the full set of 150. Main is the harder mix and is the subset Artificial Analysis runs.

Where is the FrontierCode leaderboard?

Cognition publishes one at cognition.com/frontiercode, and Artificial Analysis runs an independent version with its Stirrup harness. Since October 23, 2026, FrontierCode is also one third of the Artificial Analysis Coding Agent Index v2.0.

Is FrontierCode better than SWE-bench?

It measures something SWE-bench Verified doesn't: whether a passing patch is good enough to merge. Cognition claims 81% fewer grading errors than SWE-Bench Pro, but the tasks are private, so that claim can't yet be checked independently. Use both, and read our SWE-Bench Pro guide for the public alternative.