79 passes, up from 3
Claude Haiku 4.5 passed 3 of 657 tasks (0.5%) on AutomationBench 1.0.6. Haiku 5.5 passes 79 at max effort (12.0%), and 40 even at low (6.1%).
Release report · October 2026Claude Haiku 5.5 · With default fallbacks
Haiku 4.5 passed 3 tasks. Haiku 5.5 passes 79.
Claude Haiku 5.5 completes 12.0% of AutomationBench's strict, end-to-end business tasks at max effort, up from 0.5% for Claude Haiku 4.5. Other fast, low-cost models score higher: GPT-6 Luna reaches 20.7% and Gemini 3.7 Flash 30.4%.
Best pass rate12.0%Max effort · 79 of 657 tasks
vs. Claude Haiku 4.5+11.5pts0.5% at Default
Key findings
Refusals hold the score down
Anthropic's classifier refused 13.2% of tasks at low effort and 37.0% at max. All 763 refusals were labeled biology-related, and the fallback model served no calls.
Still behind other fast models
On the leaderboard, GPT-6 Luna reaches 20.7% and Gemini 3.7 Flash 30.4%. Haiku 5.5's best is 12.0%, at max effort.
Effort buys reasoning, not steps
From low to max, reasoning tokens grow 11.1× while steps per task barely move, from 14.5 to 14.8. XHigh (9.9%) scores below High (10.5%).
01
Results by effort
Passes nearly double from low to max: 6.1% to 12.0%.
Scoring is strict. A task passes only if every assertion about the final state of the simulated company holds. All 657 tasks stay in the denominator at every effort, refusals included. The dashed line marks Claude Haiku 4.5, at 0.5%.
Official pass rate · task_completed_correctly · 657 tasks per effort, all in the denominator
02
Against the field
A big step from Haiku 4.5, still behind its peers.
Each bar is a model's best effort setting on AutomationBench 1.0.6, from the results published on zapier.com/benchmarks. Haiku 5.5 is shown at max effort; the fastest, lowest-cost models from each lab are included for comparison.
Gemini 3.7 FlashHigh
Gemini 3.8 FlashHigh
GPT-6 LunaMax
DeepSeek V4 FlashMax
Claude Haiku 5.5Max
Claude Haiku 4.5Default
Best observed pass rate per model · AutomationBench 1.0.6 · Comparison results are each model's best published result on zapier.com/benchmarks (AutomationBench 1.0.6). Claude Haiku 4.5 has a single published setting. Costs are withheld: this run was priced at reference rates, not Haiku 5.5 pricing.
03
By domain
Marketing and Operations reach 16.0%. HR is the hardest, at 9.2%.
Sales gains the most from effort, 10.2 points from low to max, and Finance 10.0. HR peaks at medium and finishes lower at max (5.0%).
0%10%20%30%
Low → MaxMarketing
9.0%16.0%
+7.0ptsVentes
4.3%14.5%
+10.2ptsOpérations
12.0%13.0%
+1.0ptsFinances
2.5%12.5%
+10.0ptsAssistance
4.0%12.0%
+8.0ptsHR
5.8%5.0%
−0.8pts Low Medium, High, XHigh Max
Full matrix
| Domaine | Low | Moyen | High | XHigh | Max |
|---|---|---|---|---|---|
| Ventes | 4.3%5/117 | 6.8%8/117 | 7.7%9/117 | 9.4%11/117 | 14.5%17/117 |
| Marketing | 9.0%9/100 | 8.0%8/100 | 15.0%15/100 | 14.0%14/100 | 16.0%16/100 |
| Opérations | 12.0%12/100 | 15.0%15/100 | 16.0%16/100 | 13.0%13/100 | 13.0%13/100 |
| Assistance | 4.0%4/100 | 7.0%7/100 | 9.0%9/100 | 8.0%8/100 | 12.0%12/100 |
| Finances | 2.5%3/120 | 5.8%7/120 | 10.0%12/120 | 7.5%9/120 | 12.5%15/120 |
| HR | 5.8%7/120 | 9.2%11/120 | 6.7%8/120 | 8.3%10/120 | 5.0%6/120 |
Pass rate · passed / tasks per cell · darker = higher
04
Tokens & tools
Reasoning grows 11.1×. Steps stay flat.
From low to max, reasoning tokens per task rise from 4,663 to 51,833 and tool calls by 27%, while steps hold near 15. About 86 to 89% of input is served from cache, and no task hit the 50-step cap.
Reasoning Visible output (text + tool calls)Mean output tokens per task
| Effort | Tool calls | Étapes | Calls / step | Input tokens | Cached share | At step cap |
|---|---|---|---|---|---|---|
| Low | 48.3 | 14.5 | 3.3 | 388.2k | 88.4% | 0 |
| Moyen | 53.1 | 15.3 | 3.5 | 457.3k | 88.7% | 0 |
| High | 56.5 | 15.5 | 3.6 | 519.2k | 88.3% | 0 |
| XHigh | 59.4 | 15.3 | 3.9 | 577.0k | 87.4% | 0 |
| Max | 61.3 | 14.8 | 4.1 | 755.3k | 85.5% | 0 |
Mean per task, all tasks included · step cap 50
05
Refusals
Refusals climb from 13.2% at low to 37.0% at max.
A refused task counts as a failure unless its end state had already passed; 50 of 763 blocked tasks had, and count as passes. Operations draws the most, 58.0% of tasks at max. Every refusal was labeled biology-related by Anthropic's classifier, and those refusals aren't eligible for the fallback model.
13.2%
Low87/65717.0%
Medium112/65721.5%
High141/65727.4%
XHigh180/65737.0%
Max243/657| Domaine | Low | Moyen | High | XHigh | Max |
|---|---|---|---|---|---|
| Ventes | 15.4% | 21.4% | 25.6% | 32.5% | 39.3% |
| Marketing | 15.0% | 24.0% | 27.0% | 39.0% | 51.0% |
| Opérations | 35.0% | 41.0% | 47.0% | 53.0% | 58.0% |
| Assistance | 4.0% | 6.0% | 15.0% | 15.0% | 28.0% |
| Finances | 5.8% | 5.8% | 5.0% | 13.3% | 24.2% |
| HR | 6.7% | 7.5% | 13.3% | 15.8% | 25.8% |
Recorded refusal flags · refused tasks stay in the denominator
06
Methodology
AutomationBench tests AI agents on end-to-end workflow execution using 47 simulated apps across six business functions: Sales, Marketing, Operations, Support, Finance and HR. Each task boots a small simulated company, hands the agent one request a real operations person would get, then grades the world it leaves behind with deterministic assertions.
- 01
Boot a tiny company
CRM records, inbox threads, spreadsheets and support cases, including traps like stale rows and near-duplicate names.
- 02
The agent works alone
The agent gets one trigger message and tools to search, read, update and send. It can't ask clarifying questions, and there's no human in the loop.
- 03
Grade the end state
Deterministic assertions check what changed. The task passes only if every one holds.
ComparisonsComparison results are each model's best published result on zapier.com/benchmarks (AutomationBench 1.0.6). Claude Haiku 4.5 has a single published setting. Costs are withheld: this run was priced at reference rates, not Haiku 5.5 pricing.