🔍 Read the full analysis: FrontierCode: What Helps An AI Pull Request Make A Strong Case? on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added the repository-focused FrontierCode benchmark to its evaluations on October 12, 2026, testing 23 models. Claude Opus 5.5 led with a 52.2% score and cleared every blocking check in 57.7% of attempts, while results also showed large differences in cost and performance.
Artificial Analysis added FrontierCode to its model evaluations on October 12, reporting results for 23 coding models on tasks designed to test whether a pull request would meet maintainers’ standards. Claude Opus 5.5 (max with fallback) ranked first with a 52.2% benchmark score, but it cleared all blocking checks in 57.7% of attempts, leaving substantial room for human review.
FrontierCode evaluates repository changes on more than whether tests pass. Its checks cover hidden tests, command checks, reverse tests that test whether the agent’s tests fail on the original code, adaptive tests, limits on file scope and diff size, and an LLM code review against maintainer criteria. An attempt that fails any blocking check receives a zero; passing all blocking checks produces a score from 70% to 100%, depending on quality checks. Cognition’s trajectory scanner also gives a zero to attempts flagged for looking up solutions.
In Artificial Analysis’s results, GPT-6.1 Sol (max) scored 49.1% and ranked second, followed by Claude Fable 5.1 at 49.0%. The reported pass rate—the share of attempts clearing every blocking check—was 54.7% for GPT-6.1 Sol and 54.3% for Fable 5.1. Nine of the benchmark’s 100 Main-set tasks were not solved by any model on any attempt, according to the source report.
Cost varied widely. Artificial Analysis reported $10.15 per task for Opus and $0.81 for GPT-6.1 Sol; it described the latter as a 12.5-times lower cost, for a 3.1-point lower benchmark score. The listed cost frontier also included GPT-6 Luna (max), scoring 41.9% at $0.09 per task, and MiMo-V2.6-Pro, scoring 45.4% at $0.18. These figures are benchmark-specific costs, not estimates of the full cost of deploying or reviewing a coding system.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
What the Scores Say About Review
The results measure a more demanding target than basic code generation: whether an agent can make a bounded, tested, reviewable change in an existing repository. That makes the benchmark relevant to software teams considering coding agents, because a patch that passes tests can still be too broad, insufficiently tested, or inconsistent with maintainers’ expectations.
The leading model’s 57.7% blocking-check pass rate suggests that even a top result does not support treating autonomous pull requests as dependable without oversight. That is an interpretation of this benchmark, not a direct measure of performance in every company’s codebase. FrontierCode’s tasks and scoring rules set a specific standard, and actual outcomes may vary by repository, workflow, and review practice.
Cost adds another practical trade-off. Sol’s score was 3.1 points below Opus’s at a reported $0.81 per task rather than $10.15. Teams comparing models may need to weigh that price difference against the chance of clearing the benchmark’s blocking checks, along with the time people spend correcting and reviewing failures. The benchmark does not establish which option will be more economical in a particular production setting.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Repository Changes
Cognition developed FrontierCode with open-source maintainers. The benchmark was announced in June 2026, and version 1.1 followed in July with rules against searching online for task solutions. According to the source report, more than 20 maintainers wrote tasks drawn from 36 well-known repositories, including Celery, Budibase, uppy and Mattermost, spending more than 40 hours on each task.
The full benchmark contains 150 tasks; Artificial Analysis ran the 100-task Main set. Tasks are private so that models cannot have encountered the solutions in training, according to the report. For its evaluation, Artificial Analysis used the same open-source agent harness, Stirrup, for each model, with a cap of 250 turns per attempt and three attempts per task. GPT-6 Luna (medium) handled the code-review judging step.
The benchmark’s design makes passing tests only part of the evaluation. Reverse tests check whether an agent’s tests would catch the original defect, while scope rules penalize changes outside relevant files or diffs that are too large. Tomer Nosrati, a Celery maintainer, described the approach in Cognition’s announcement as: “Where others grade like a CI, FrontierCode grades like a tech lead.”
““Where others grade like a CI, FrontierCode grades like a tech lead.””
— Tomer Nosrati, Celery maintainer, in Cognition’s announcement
automated pull request review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Available Results
The results do not show how every model would perform in a specific organization’s repositories or how much human review each generated change would require. The source report gives benchmark scores and per-task costs, but does not provide a direct measure of production reliability, review time, or the cost of correcting unsuccessful attempts.
Several models had no FrontierCode result as of October 13, according to the report, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their performance cannot be inferred from results on other coding evaluations. The report also does not establish whether subsequent benchmark runs will use the same models, task set, or evaluation configuration.
software development code analysis tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Further Model Results to Watch
The next relevant development is whether models absent from the October 13 comparison receive FrontierCode results, and whether later evaluations change the standings. Any new result should be read alongside its model version, evaluation settings and benchmark task set, rather than treated as directly interchangeable if those details change.
For developers and maintainers, the immediate use of this report is comparative: it offers evidence about how tested models handle one set of repository tasks under a shared harness. Human review remains part of the process in interpreting these results, since the benchmark does not establish that any model can safely merge changes without oversight.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated changes to real software repositories using checks for correctness, regression risk, test quality, change scope and maintainer-style review criteria. A passing test suite alone does not guarantee a high score.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s reported evaluation, with a 52.2% FrontierCode score and a 57.7% pass rate on blocking checks.
Does the leading score mean AI pull requests are ready to merge automatically?
No. The reported pass rate means the leading model cleared every blocking check in 57.7% of attempts on this benchmark. It does not establish performance across all repositories or remove the need for review.
How do model costs compare in the report?
Artificial Analysis reported $0.09 per task for GPT-6 Luna (max), $0.81 for GPT-6.1 Sol (max), and $10.15 for Claude Opus 5.5 (max with fallback). These are benchmark-specific figures, not guarantees of costs in other deployments.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
