Medium beats last generation's max
At medium effort, Opus 5.5 passes 28.6% of tasks for $2.33 each. Opus 5 at max passes 26.9% for $3.98. Higher accuracy for 41% less.
Release report · September 2026Claude Opus 5.5 · Without fallbacks
A generation's jump, at last generation's price.
Claude Opus 5.5 completes 40.0% of AutomationBench's strict, end-to-end business tasks at max effort. That's 13.1 points above Opus 5 for 6% more per task, and 1.4 points behind GPT-6 Astra for 17% less.
Best pass rate40.0%Max effort · 263 of 657 tasks
vs. Claude Opus 5+13.1pts26.9% at Max
Cost per task$4.22Max effort · no-cache estimate
On the frontier5/ 5effort settings no model in this comparison beats on both cost and accuracy
Key findings
Every effort setting sits on the frontier
Of the 25 model-and-effort configurations in this run's comparison, eight are Pareto-optimal: nothing else is both cheaper and more accurate. Opus 5.5 holds five of them, one at every effort level. Claude Sonnet 5.5, tested a week later, isn't part of this comparison.
Effort pays off unevenly
Support gains the most, from 15% at low to 36% at max. Operations leads at 62%. HR is still the hardest domain, topping out at 24.2%.
Max effort thinks more, not longer
Reasoning tokens grow 18.6× from low to max, but steps per task only rise from 16.1 to 18.7.
01
Results by effort
Every step up in effort buys accuracy. The biggest jump is the last one: +5.6 points from xhigh to max.
Scoring is strict. A task passes only if every assertion about the final state of the simulated company holds. All 657 tasks stay in the denominator at every effort, refusals included.
Official pass rate · task_completed_correctly · 657 tasks per effort, all in the denominator
02
Cost vs. accuracy
Medium, high and xhigh all beat Opus 5's max on both cost and accuracy.
Each line is one model, traced from low to max effort. The dotted line marks the frontier, the configurations no other model beats on both cost and accuracy. Past 40%, only GPT-6 Astra at max goes further: 41.4% at $5.10 per task.
Claude Opus 5.5 Frontier Frontier point DominatedHover a point for details
View the data
| Model | Low | Medium | High | XHigh | Max |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 23.3%$1.62 | 28.6%$2.33 | 32.0%$2.57 | 34.4%$3.17 | 40.0%$4.22 |
| Claude Fable 5.1 | 16.6%$4.74 | 17.8%$5.13 | 19.6%$5.34 | 21.3%$6.53 | 22.4%$7.09 |
| Claude Opus 5 | 20.4%$2.86 | 23.9%$3.22 | 20.5%$3.13 | 25.3%$3.68 | 26.9%$3.98 |
| GPT-6 Astra | 30.3%$3.65 | 34.1%$4.04 | 37.1%$4.48 | 39.0%$4.65 | 41.4%$5.10 |
| GPT-5.6 Sol | 11.7%$1.33 | 19.6%$1.84 | 24.8%$2.04 | 26.3%$2.37 | 28.8%$3.06 |
API cost per task, no-cache estimate · pass rate at each effort, low to max · Comparison models were run on the same benchmark version and shown at five effort levels each.
03
By domain
Operations is closest to solved. HR is still the hardest domain in the benchmark.
Support gains the most from extra effort, climbing 21 points. Sales climbs slowly until max, then jumps 11.1 points in a single step.
0%10%20%30%40%50%60%70%
Low → MaxOperations
48.0%62.0%
+14.0ptsMarketing
28.0%43.0%
+15.0ptsSales
23.9%42.7%
+18.8ptsSupport
15.0%36.0%
+21.0ptsFinance
17.5%35.8%
+18.3ptsHR
10.8%24.2%
+13.4pts Low Medium, High, XHigh Max
Full matrix
| Domain | Low | Medium | High | XHigh | Max |
|---|---|---|---|---|---|
| Sales | 23.9%28/117 | 26.5%31/117 | 28.2%33/117 | 31.6%37/117 | 42.7%50/117 |
| Marketing | 28.0%28/100 | 37.0%37/100 | 39.0%39/100 | 40.0%40/100 | 43.0%43/100 |
| Operations | 48.0%48/100 | 53.0%53/100 | 51.0%51/100 | 54.0%54/100 | 62.0%62/100 |
| Support | 15.0%15/100 | 22.0%22/100 | 33.0%33/100 | 34.0%34/100 | 36.0%36/100 |
| Finance | 17.5%21/120 | 23.3%28/120 | 30.0%36/120 | 31.7%38/120 | 35.8%43/120 |
| HR | 10.8%13/120 | 14.2%17/120 | 15.0%18/120 | 19.2%23/120 | 24.2%29/120 |
Pass rate · passed / tasks per cell · darker = higher
04
Cost
A successful task at max effort costs $10.54 once you count the failures.
Failed attempts still cost money. Spend per pass divides every task's cost by the number of passes. Max effort delivers 72% more passes than low for 52% more per pass.
| Effort | Cost / task | Passed task | Failed task | Spend per pass |
|---|---|---|---|---|
| Low | $1.62 | $2.34 | $1.40 | $6.95 |
| Medium | $2.33 | $3.06 | $2.05 | $8.16 |
| High | $2.57 | $3.45 | $2.16 | $8.05 |
| XHigh | $3.17 | $4.14 | $2.66 | $9.21 |
| Max | $4.22 | $5.04 | $3.67 | $10.54 |
USD, no-cache estimate · $4.00 input / $20.00 output per 1M tokens · spend per pass = all task costs ÷ passes, failures included
05
Tokens & tools
Reasoning grows 18.6×. Steps barely move.
From low to max, tool calls per task rise 41% while steps rise only 16%. Higher effort packs more thinking and more calls into each step instead of running longer. About 90% of input is served from cache at every level, and no task hit the 50-step cap.
Reasoning Visible output (text + tool calls)Mean output tokens per task
| Effort | Tool calls | Steps | Calls / step | Input tokens | Cached share | At step cap |
|---|---|---|---|---|---|---|
| Low | 46.8 | 16.1 | 2.9 | 341.0k | 89.1% | 0 |
| Medium | 52.1 | 17.9 | 2.9 | 505.1k | 90.6% | 0 |
| High | 53.9 | 18.3 | 2.9 | 557.6k | 90.9% | 0 |
| XHigh | 58.9 | 19.3 | 3.1 | 682.6k | 91.3% | 0 |
| Max | 65.8 | 18.7 | 3.5 | 856.9k | 90.1% | 0 |
Mean per task, all tasks included · step cap 50
06
Refusals
Refusals roughly double at max effort, from 2.0% to 4.4%.
A refused task counts as a failure. Marketing draws the most refusals, 10% of tasks at max. These are recorded refusal flags, not inferred from response text.
2.0%
Low13/6572.0%
Medium13/6572.1%
High14/6573.0%
XHigh20/6574.4%
Max29/657| Domain | Low | Medium | High | XHigh | Max |
|---|---|---|---|---|---|
| Sales | 1.7% | 0.9% | 1.7% | 5.1% | 4.3% |
| Marketing | 7.0% | 7.0% | 4.0% | 6.0% | 10.0% |
| Operations | 1.0% | 2.0% | 2.0% | 3.0% | 4.0% |
| Support | 1.0% | 0.0% | 1.0% | 2.0% | 4.0% |
| Finance | 0.8% | 0.0% | 1.7% | 0.8% | 1.7% |
| HR | 0.8% | 2.5% | 2.5% | 1.7% | 3.3% |
Recorded refusal flags · refused tasks stay in the denominator
07
Methodology
AutomationBench tests AI agents on end-to-end workflow execution using 47 simulated apps across six business functions: Sales, Marketing, Operations, Support, Finance and HR. Each task boots a small simulated company, hands the agent one request a real operations person would get, then grades the world it leaves behind with deterministic assertions.
- 01
Boot a tiny company
CRM records, inbox threads, spreadsheets and support cases, including traps like stale rows and near-duplicate names.
- 02
The agent works alone
The agent gets one trigger message and tools to search, read, update and send. It can't ask clarifying questions, and there's no human in the loop.
- 03
Grade the end state
Deterministic assertions check what changed. The task passes only if every one holds.
CostCosts are a no-cache estimate at the provider's announced launch rates. Every input token is billed at the full rate, cache discounts are ignored and reasoning is billed as output. These are not billed costs, and they run higher than the leaderboard's Cost / task, so don't compare the two directly.
Rates$4.00 input · $20.00 output · $0.20 cache read per 1M tokens.
ComparisonsComparison models were run on the same benchmark version and shown at five effort levels each.