Medium effort already clears 50%
At medium, Gemini 4 Argon passes 329 of 657 tasks (50.1%). High adds 8 more, for 51.3%. Every other model on the leaderboard sits below 45%.
Release report · September 2026Gemini 4 Argon
Past halfway, for the first time.
Gemini 4 Argon completes 51.3% of AutomationBench's strict, end-to-end business tasks at high effort, the first result above 50% on the benchmark. That's 6.6 points above Claude Sonnet 5.5, the next-best model on the leaderboard. Medium effort alone reaches 50.1%, at $1.54 a task at Google's list rates.
Best pass rate51.3%High effort · 337 of 657 tasks
vs. Claude Sonnet 5.5+6.6pts44.7% at Max
Cost per task$1.70High effort · list-price estimate
Domains led5/ 6best effort per domain, vs. the best other model
On the frontier2/ 2effort settings no model in this comparison beats on both cost and accuracy
Key findings
Ahead in five of six domains
Taking each domain at its best effort setting, Gemini 4 Argon beats every other model's best in Sales (58.1% at medium vs. 45.3%), Operations (68.0% at medium vs. 65.0%), Marketing, Support and HR. Finance is the exception: 49.2% vs. Claude Sonnet 5.5's 50.0%.
More effort isn't better everywhere
High beats medium in Support (+7.0 points), Marketing (+5.0) and HR (+3.4). Medium does better in Sales, Operations and Finance.
8.7 points more, for 11% less
At list rates, Gemini 4 Argon costs $1.54 a task at medium for 50.1%. GPT-6 Astra at max costs $1.73 for 41.4%.
01
Results by effort
Medium does nearly all of the work: 50.1%, against 51.3% at high.
Scoring is strict. A task passes only if every assertion about the final state of the simulated company holds. All 657 tasks stay in the denominator at both settings, refusals included. The dashed line marks Claude Sonnet 5.5 at max effort, the best result from any other model on the leaderboard.
Official pass rate · task_completed_correctly · 657 tasks per effort, all in the denominator
02
Against the field
A 6.6-point lead over the next-best model.
Each bar is a model's best effort setting on AutomationBench 1.0.6. Gemini 4 Argon is shown at high effort; the other results are the ones published on zapier.com/benchmarks. The Claude models appear only here, since their published costs use a different basis.
Gemini 4 ArgonHigh
Claude Sonnet 5.5Max
Claude Opus 5.5Max
GPT-6 AstraMax
GPT-6 SolXHigh
Gemini 3.8 FlashMedium
GPT-6 LunaMax
Best observed pass rate per model · AutomationBench 1.0.6
03
Cost vs. accuracy
Medium beats GPT-6 Astra's best by 8.7 points, for 11% less per task.
Each line is one model's effort settings, pass rate against cost per task. Both Gemini 4 Argon settings sit on the frontier: no configuration in the comparison is both cheaper and more accurate. The Claude models are left out here because their published costs use a different basis.
Gemini 4 Argon Frontier Frontier point DominatedHover a point for details
View the data
| Model | Medium | High |
|---|---|---|
| Gemini 4 Argon | 50.1%$1.54 | 51.3%$1.70 |
| Gemini 3.8 Flash | 29.7%$0.77 | 29.7%$0.91 |
| GPT-6 Sol | 26.9%$0.21 | 31.2%$0.24 |
| GPT-6 Astra | 34.1%$1.27 | 37.1%$1.44 |
| GPT-6 Luna | 9.4%$0.02 | 14.5%$0.02 |
API cost per task, list-price estimate · pass rate at each effort, low to max · Cost comparisons use the models in this run's metrics export. Claude Sonnet 5.5 and Claude Opus 5.5 are pass-rate results from the zapier.com/benchmarks leaderboard, with Anthropic's default fallbacks.
04
By domain
Operations reaches 68.0% at medium. HR is still the hardest domain, at 39.2%.
With two effort settings, each line runs from medium to high. Support gains the most from extra effort, 7 points. Sales and Operations do better at medium.
0%10%20%30%40%50%60%70%80%
Medium → HighOperations
68.0%64.0%
−4.0ptsMarketing
53.0%58.0%
+5.0ptsSales
58.1%55.6%
−2.5ptsFinance
49.2%48.3%
−0.9ptsSupport
38.0%45.0%
+7.0ptsHR
35.8%39.2%
+3.4pts Medium High
Full matrix
| Domain | Medium | High |
|---|---|---|
| Sales | 58.1%68/117 | 55.6%65/117 |
| Marketing | 53.0%53/100 | 58.0%58/100 |
| Operations | 68.0%68/100 | 64.0%64/100 |
| Support | 38.0%38/100 | 45.0%45/100 |
| Finance | 49.2%59/120 | 48.3%58/120 |
| HR | 35.8%43/120 | 39.2%47/120 |
Pass rate · passed / tasks per cell · darker = higher
05
Cost
$1.54 a task at medium, $1.70 at high.
High costs 11% more per task for 1.2 more points, so spend per passed task rises 8%, from $3.07 to $3.32. Spend per pass counts the cost of every failed task too. These are estimates at announced list rates, not billed costs; Google's 50% launch promotion halves them.
| Effort | Cost / task | Passed task | Failed task | Spend per pass |
|---|---|---|---|---|
| Medium | $1.54 | $1.38 | $1.69 | $3.07 |
| High | $1.70 | $1.51 | $1.91 | $3.32 |
USD, list-price estimate · spend per pass = all task costs ÷ passes, failures included
06
Tokens & tools
About five tool calls per step, at both settings.
High effort uses 22% more reasoning tokens than medium (25.8k vs. 21.1k per task) and 0.7 more steps. About 60% of input is served from cache, and one task per setting hit the 50-step cap.
Reasoning Visible output (text + tool calls)Mean output tokens per task
| Effort | Tool calls | Steps | Calls / step | Input tokens | Cached share | At step cap |
|---|---|---|---|---|---|---|
| Medium | 69.9 | 13.3 | 5.3 | 497.7k | 59.2% | 1 |
| High | 71.4 | 14.0 | 5.1 | 575.6k | 62.7% | 1 |
Mean per task, all tasks included · step cap 50
07
Refusals
Three refusals in 1,314 task runs.
A refused task counts as a failure. Support and Finance each drew one at medium, and Operations one at high. These are recorded refusal flags, not inferred from response text.
0.3%
Medium2/6570.2%
High1/657| Domain | Medium | High |
|---|---|---|
| Sales | 0.0% | 0.0% |
| Marketing | 0.0% | 0.0% |
| Operations | 0.0% | 1.0% |
| Support | 1.0% | 0.0% |
| Finance | 0.8% | 0.0% |
| HR | 0.0% | 0.0% |
Recorded refusal flags · refused tasks stay in the denominator
08
Methodology
AutomationBench tests AI agents on end-to-end workflow execution using 47 simulated apps across six business functions: Sales, Marketing, Operations, Support, Finance and HR. Each task boots a small simulated company, hands the agent one request a real operations person would get, then grades the world it leaves behind with deterministic assertions.
- 01
Boot a tiny company
CRM records, inbox threads, spreadsheets and support cases, including traps like stale rows and near-duplicate names.
- 02
The agent works alone
The agent gets one trigger message and tools to search, read, update and send. It can't ask clarifying questions, and there's no human in the loop.
- 03
Grade the end state
Deterministic assertions check what changed. The task passes only if every one holds.
CostEstimates at Google's announced list rates for Gemini 4 Argon: $4.00 input and $20.00 output per 1M tokens, with reasoning billed as output. No cache-read price has been announced, so cache reads are assumed at $0.40 per 1M (10% of input, Gemini's usual discount). These are not billed costs, and Google's 50% launch promotion halves them.
ComparisonsCost comparisons use the models in this run's metrics export. Claude Sonnet 5.5 and Claude Opus 5.5 are pass-rate results from the zapier.com/benchmarks leaderboard, with Anthropic's default fallbacks.