Ahead of Opus 5.5 only at the top two settings
At xhigh, Sonnet 5.5 passes 36.8% of tasks for $0.45 each; Opus 5.5 passes 34.4% for $0.80. At max it's 44.7% vs. 40.0% for 16% less. Below xhigh, Opus 5.5 leads by 2.6 to 5.3 points.
Release report · September 2026Claude Sonnet 5.5 · With default fallbacks
At full effort, Sonnet catches Opus.
Claude Sonnet 5.5 completes 44.7% of AutomationBench's strict, end-to-end business tasks at max effort. That's 4.7 points above Claude Opus 5.5 without fallbacks, for 16% less per task. Opus 5.5 still leads at low, medium and high effort.
Best pass rate44.7%Max effort · 294 of 657 tasks
vs. Claude Opus 5.5+4.7pts40.0% at Max
Cost per task$1.08Max effort · list-price estimate
On the frontier2/ 6effort settings no model in this comparison beats on both cost and accuracy
Key findings
Each step up buys more than the last
Gains grow from +5.0 points into medium to +6.1, +7.4 and +7.9 on the way to max. Low effort (18.3%) does no better than the Between tools setting (18.6%).
The cheapest pass is at high, not max
Counting failures, a completed task costs $1.15 at high and $2.42 at max, 110% more. Max buys 101 more passes than high: 294 vs. 193 of 657.
Max is a different mode
Reasoning tokens grow 6.5× from xhigh to max alone, and cost per task 2.4×. Every setting below max stays under $0.46 per task.
01
Results by effort
The biggest jump is the last one: +7.9 points from xhigh to max.
Scoring is strict. A task passes only if every assertion about the final state of the simulated company holds. All 657 tasks stay in the denominator at every effort, refusals included. Default fallbacks were on for every request; the fallback model answered one task, which still failed, so fallbacks added no passes.
Official pass rate · task_completed_correctly · 657 tasks per effort, all in the denominator
02
Cost vs. accuracy
Sonnet 5.5 is the cheapest way to reach any score from 33.2% to 44.7%.
Each line is one model, traced from its lowest to its highest effort. The dotted line marks the frontier, the configurations no other model beats on both cost and accuracy. Below 33.2%, GPT-6 Sol is cheaper at every level, from $0.19 per task.
Claude Sonnet 5.5 Frontier Frontier point DominatedHover a point for details
View the data
| Mannequin | Between tools | Low | Moyen | High | XHigh | Max |
|---|---|---|---|---|---|---|
| Claude Sonnet 5.5 | 18.6%$0.28 | 18.3%$0.26 | 23.3%$0.29 | 29.4%$0.34 | 36.8%$0.45 | 44.7%$1.08 |
| Claude Opus 5.5 | — | 23.3%$0.46 | 28.6%$0.60 | 32.0%$0.65 | 34.4%$0.80 | 40.0%$1.28 |
| Claude Opus 5 | — | 20.4%$0.35 | 23.9%$0.41 | 20.5%$0.44 | 25.3%$0.53 | 26.9%$0.60 |
| GPT-6 Sol | — | 21.2%$0.19 | 26.9%$0.21 | 31.2%$0.24 | 33.2%$0.27 | 32.0%$0.34 |
API cost per task, list-price estimate · pass rate at each effort, low to max · Comparison models were run on the same benchmark version at five effort levels each; Sonnet 5.5 adds a sixth, Between tools.
03
By domain
Finance and Marketing gain the most from effort. HR is still the hardest domain.
Finance climbs from 17.5% at Between tools to 50.0% at max, and Marketing from 20.0% to 52.0%. Operations leads at every effort and peaks at xhigh (60.0%), not max (59.0%).
0%10%20%30%40%50%60%70%
Between tools → MaxOpérations
34.0%59.0%
+25.0ptsMarketing
20.0%52.0%
+32.0ptsFinances
17.5%50.0%
+32.5ptsVentes
15.4%41.9%
+26.5ptsAssistance
12.0%36.0%
+24.0ptsHR
14.2%31.7%
+17.5pts Between tools Low, Medium, High, XHigh Max
Full matrix
| Domaine | Between tools | Low | Moyen | High | XHigh | Max |
|---|---|---|---|---|---|---|
| Ventes | 15.4%18/117 | 17.1%20/117 | 19.7%23/117 | 27.4%32/117 | 31.6%37/117 | 41.9%49/117 |
| Marketing | 20.0%20/100 | 23.0%23/100 | 28.0%28/100 | 41.0%41/100 | 45.0%45/100 | 52.0%52/100 |
| Opérations | 34.0%34/100 | 39.0%39/100 | 44.0%44/100 | 52.0%52/100 | 60.0%60/100 | 59.0%59/100 |
| Assistance | 12.0%12/100 | 13.0%13/100 | 14.0%14/100 | 21.0%21/100 | 31.0%31/100 | 36.0%36/100 |
| Finances | 17.5%21/120 | 10.0%12/120 | 24.2%29/120 | 26.7%32/120 | 39.2%47/120 | 50.0%60/120 |
| HR | 14.2%17/120 | 10.8%13/120 | 12.5%15/120 | 12.5%15/120 | 18.3%22/120 | 31.7%38/120 |
Pass rate · passed / tasks per cell · darker = higher
04
Coût
A successful task at max effort costs $2.42 once you count the failures.
Failed attempts still cost money. Spend per pass divides every task's cost by the number of passes. It bottoms out at high ($1.15), then rises 110% at max.
| Effort | Cost / task | Passed task | Failed task | Spend per pass |
|---|---|---|---|---|
| Between tools | $0.28 | $0.31 | $0.27 | $1.51 |
| Low | $0.26 | $0.34 | $0.24 | $1.41 |
| Moyen | $0.29 | $0.35 | $0.27 | $1.23 |
| High | $0.34 | $0.41 | $0.31 | $1.15 |
| XHigh | $0.45 | $0.51 | $0.42 | $1.22 |
| Max | $1.08 | $1.12 | $1.05 | $2.42 |
USD, list-price estimate · spend per pass = all task costs ÷ passes, failures included
05
Tokens & tools
Reasoning grows 88.6×. Steps grow 37%.
From Between tools to max, tool calls per task rise 51% and steps from 13.4 to 18.4, with most of the growth in the last step. About 88% of input is served from cache at every level, and one task hit the 50-step cap.
Reasoning Visible output (text + tool calls)Mean output tokens per task
| Effort | Tool calls | Étapes | Calls / step | Input tokens | Cached share | At step cap |
|---|---|---|---|---|---|---|
| Between tools | 51.8 | 13.4 | 3.9 | 357.8k | 87.7% | 0 |
| Low | 48.2 | 13.9 | 3.5 | 320.0k | 87.4% | 0 |
| Moyen | 50.6 | 14.2 | 3.6 | 359.9k | 87.7% | 0 |
| High | 54.2 | 14.4 | 3.8 | 437.4k | 88.2% | 0 |
| XHigh | 59.6 | 15.4 | 3.9 | 575.8k | 88.9% | 0 |
| Max | 78.2 | 18.4 | 4.3 | 1209.8k | 89.8% | 1 |
Mean per task, all tasks included · step cap 50
06
Refusals
Refusals stay near 1% until max, then rise to 4.9%.
A refused task counts as a failure. Operations draws the most, 13% of tasks at max. All 66 recorded refusals were labeled biology-related by Anthropic's classifier, and those refusals aren't eligible for the fallback model.
1.2%
Between tools8/6570.8%
Low5/6570.9%
Medium6/6571.1%
High7/6571.2%
XHigh8/6574.9%
Max32/657| Domaine | Between tools | Low | Moyen | High | XHigh | Max |
|---|---|---|---|---|---|---|
| Ventes | 4.3% | 3.4% | 4.3% | 4.3% | 3.4% | 9.4% |
| Marketing | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 6.0% |
| Opérations | 3.0% | 1.0% | 1.0% | 2.0% | 3.0% | 13.0% |
| Assistance | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Finances | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| HR | 0.0% | 0.0% | 0.0% | 0.0% | 0.8% | 1.7% |
Recorded refusal flags · refused tasks stay in the denominator
07
Methodology
AutomationBench tests AI agents on end-to-end workflow execution using 47 simulated apps across six business functions: Sales, Marketing, Operations, Support, Finance and HR. Each task boots a small simulated company, hands the agent one request a real operations person would get, then grades the world it leaves behind with deterministic assertions.
- 01
Boot a tiny company
CRM records, inbox threads, spreadsheets and support cases, including traps like stale rows and near-duplicate names.
- 02
The agent works alone
The agent gets one trigger message and tools to search, read, update and send. It can't ask clarifying questions, and there's no human in the loop.
- 03
Grade the end state
Deterministic assertions check what changed. The task passes only if every one holds.
CostCosts are estimates at Anthropic's announced launch rates for Sonnet 5.5: $2.00 input, $0.20 cache read and $10.00 output per 1M tokens, with reasoning billed as output. Cache reads are priced at the discounted rate and the cache-write premium is not included, so estimates run slightly low. These are not billed costs.
ComparisonsComparison models were run on the same benchmark version at five effort levels each; Sonnet 5.5 adds a sixth, Between tools.