Medium already sets a new high
At medium, Gemini 4 Carbon passes 348 of 657 tasks (53.0%), above Gemini 4 Argon's best of 51.3%, for 3% less per task than Argon at high. High reaches 354, for 53.9%.
Release report · October 2026Gemini 4 Carbon
A new high, at medium effort.
Gemini 4 Carbon completes 53.9% of AutomationBench's strict, end-to-end business tasks at high effort, 2.6 points above Gemini 4 Argon and the highest score on the benchmark so far. Medium reaches 53.0%, 0.9 points below high; low effort reaches 31.5%.
Best pass rate53.9%High effort · 354 of 657 tasks
vs. Gemini 4 Argon+2.6pts51.3% at High
Cost per task$2.12High effort · list-price estimate
Low → medium+21.5ptsmedium → high adds 0.9
On the frontier2/ 3effort settings no model in this comparison beats on both cost and accuracy
Key findings
The jump is from low to medium
Moving from low to medium adds 21.5 points, from 31.5% to 53.0%. Moving from medium to high adds 0.9.
Its widest domain lead is Finance
Taking each domain at its best effort, Gemini 4 Carbon reaches 60.0% in Finance, 10.0 points above Claude Sonnet 5.5's 50.0%, the best other result. It ties Gemini 4 Argon in Operations at 68.0% and trails it in Sales (56.4% vs. 58.1%) and HR (35.0% vs. 39.2%).
Less thinking, more steps
At low effort, Gemini 4 Carbon averages 20.5 steps per task and 39 tasks hit the 50-step cap. At high, steps fall to 16.6 and 3 tasks hit the cap, while reasoning tokens grow 10.8×.
01
Results by effort
Medium adds 21.5 points over low: 31.5% to 53.0%.
Scoring is strict. A task passes only if every assertion about the final state of the simulated company holds. All 657 tasks stay in the denominator at every effort. The dashed line marks Gemini 4 Argon at high effort, the best result from any other model on the leaderboard.
Official pass rate · task_completed_correctly · 657 tasks per effort, all in the denominator
02
Against the field
2.6 points above Gemini 4 Argon, the previous high.
Each bar is a model's best effort setting on AutomationBench 1.0.6. Gemini 4 Carbon is shown at high effort; the other results are the ones published on zapier.com/benchmarks. The Claude models appear only here, since their published costs use a different basis.
Gemini 4 CarbonHigh
Gemini 4 ArgonHigh
Claude Sonnet 5.5Max
Claude Opus 5.5Max
GPT-6 AstraMax
GPT-6 SolXHigh
Gemini 3.8 FlashMedium
GPT-6 LunaMax
Best observed pass rate per model · AutomationBench 1.0.6
03
Cost vs. accuracy
Medium beats Gemini 4 Argon's best for 3% less per task.
Each line is one model's effort settings, pass rate against cost per task. Medium and high sit on the frontier: no configuration in the comparison is both cheaper and more accurate. Low doesn't, since GPT-6 Sol at xhigh scores 33.2% for $0.27 a task. The Claude models are left out here because their published costs use a different basis.
Gemini 4 Carbon Frontier Frontier point DominatedHover a point for details
View the data
| Model | Low | Medium | High |
|---|---|---|---|
| Gemini 4 Carbon | 31.5%$1.02 | 53.0%$1.65 | 53.9%$2.12 |
| Gemini 4 Argon | — | 50.1%$1.54 | 51.3%$1.70 |
| Gemini 3.8 Flash | — | 29.7%$0.77 | 29.7%$0.91 |
| GPT-6 Sol | 21.2%$0.19 | 26.9%$0.21 | 31.2%$0.24 |
| GPT-6 Astra | 30.3%$1.08 | 34.1%$1.27 | 37.1%$1.44 |
| GPT-6 Luna | 1.2%$0.01 | 9.4%$0.02 | 14.5%$0.02 |
API cost per task, list-price estimate · pass rate at each effort, low to max · Cost comparisons use the models in this run's metrics export. Claude Sonnet 5.5 and Claude Opus 5.5 are pass-rate results from the zapier.com/benchmarks leaderboard, with Anthropic's default fallbacks.
04
By domain
Every domain's biggest gain comes from low to medium.
Each line runs across the three effort settings. Operations gains the most, 35.0 points from low to high, and reaches 68.0%. HR is still the hardest domain: it peaks at 35.0% at medium and finishes at 33.3% at high.
0%10%20%30%40%50%60%70%80%
Low → HighOperations
33.0%68.0%
+35.0ptsMarketing
33.0%62.0%
+29.0ptsFinance
35.8%60.0%
+24.2ptsSales
38.5%56.4%
+17.9ptsSupport
32.0%46.0%
+14.0ptsHR
17.5%33.3%
+15.8pts Low Medium High
Full matrix
| Domain | Low | Medium | High |
|---|---|---|---|
| Sales | 38.5%45/117 | 50.4%59/117 | 56.4%66/117 |
| Marketing | 33.0%33/100 | 64.0%64/100 | 62.0%62/100 |
| Operations | 33.0%33/100 | 65.0%65/100 | 68.0%68/100 |
| Support | 32.0%32/100 | 46.0%46/100 | 46.0%46/100 |
| Finance | 35.8%43/120 | 60.0%72/120 | 60.0%72/120 |
| HR | 17.5%21/120 | 35.0%42/120 | 33.3%40/120 |
Pass rate · passed / tasks per cell · darker = higher
05
Cost
$1.65 a task at medium, $2.12 at high.
Medium has the lowest spend per passed task, $3.11, against $3.25 at low and $3.94 at high. Spend per pass counts the cost of every failed task too. These are estimates at list rates, not billed costs.
| Effort | Cost / task | Passed task | Failed task | Spend per pass |
|---|---|---|---|---|
| Low | $1.02 | $0.94 | $1.06 | $3.25 |
| Medium | $1.65 | $1.48 | $1.84 | $3.11 |
| High | $2.12 | $1.79 | $2.51 | $3.94 |
USD, list-price estimate · spend per pass = all task costs ÷ passes, failures included
06
Tokens & tools
Low effort runs longer: 20.5 steps per task, against 16.6 at high.
At low, 39 of 657 tasks hit the 50-step cap; at medium and high, 3 each. Reasoning tokens grow 10.8× from low to high (4,448 to 48,195 per task), while tool calls barely move (95.3 at low, 93.1 at high). About 86 to 90% of input is served from cache.
Reasoning Visible output (text + tool calls)Mean output tokens per task
| Effort | Tool calls | Steps | Calls / step | Input tokens | Cached share | At step cap |
|---|---|---|---|---|---|---|
| Low | 95.3 | 20.5 | 4.6 | 944.4k | 90.1% | 39 |
| Medium | 89.0 | 16.4 | 5.4 | 932.2k | 86.8% | 3 |
| High | 93.1 | 16.6 | 5.6 | 1045.7k | 86.3% | 3 |
Mean per task, all tasks included · step cap 50
07
Methodology
AutomationBench tests AI agents on end-to-end workflow execution using 47 simulated apps across six business functions: Sales, Marketing, Operations, Support, Finance and HR. Each task boots a small simulated company, hands the agent one request a real operations person would get, then grades the world it leaves behind with deterministic assertions.
- 01
Boot a tiny company
CRM records, inbox threads, spreadsheets and support cases, including traps like stale rows and near-duplicate names.
- 02
The agent works alone
The agent gets one trigger message and tools to search, read, update and send. It can't ask clarifying questions, and there's no human in the loop.
- 03
Grade the end state
Deterministic assertions check what changed. The task passes only if every one holds.
CostEstimates at Google's list rates for Gemini 4 Carbon, the same as Gemini 4 Argon's: $4.00 input and $20.00 output per 1M tokens, with reasoning billed as output. Cache reads are assumed at $0.40 per 1M (10% of input), as in the Gemini 4 Argon report. These are not billed costs.
ComparisonsCost comparisons use the models in this run's metrics export. Claude Sonnet 5.5 and Claude Opus 5.5 are pass-rate results from the zapier.com/benchmarks leaderboard, with Anthropic's default fallbacks.