Release report · November 2026Gemini 4 Hydrogen
Gemini 4 Hydrogen completes 51.1% of AutomationBench's strict, end-to-end business tasks at high effort, for $0.82 a task. That is within 0.2 points of Gemini 4 Argon at half the cost per task, and 2.8 points below Gemini 4 Carbon, the highest score on the benchmark.
At high effort Gemini 4 Hydrogen passes 336 of 657 tasks for $0.82 each, against $1.63 for Gemini 4 Argon's 51.3% and $1.94 for Gemini 4 Carbon's 53.9%. It scores 2.8 points below Carbon and costs 58% less per task.
No configuration in this comparison is both cheaper and more accurate than any of Hydrogen's three. Medium is the value setting: 46.7% for $0.41 a task, half the cost of high for 4.4 points less.
Sales climbs 36.7 points from low to high, 15.4% to 52.1%, the widest effort gain of any domain. Operations behaves the other way: it peaks at medium, 65.0%, and slips to 64.0% at high.
No task hits the 50-step cap at any effort, 0 of 657 all three times. Tool calls more than double from low to high (53.4 to 116.4) and reasoning tokens grow 8.7× (5,001 to 43,666 per task), while cached input rises from 80.4% to 90.0%.
Medium adds 16.1 points over low: 30.6% to 46.7%.
Scoring is strict. A task passes only if every assertion about the final state of the simulated company holds. All 657 tasks stay in the denominator at every effort. The dashed line marks Gemini 4 Argon at high effort, the strongest other result in this comparison.
Official pass rate · task_completed_correctly · 657 tasks per effort, all in the denominator
Within 0.2 points of Gemini 4 Argon, for half the price.
Each bar is a model's best effort setting on AutomationBench 1.0.6. Gemini 4 Hydrogen is shown at high effort; the other results are the ones published on zapier.com/benchmarks. The Claude models appear only here, since their published costs use a different basis.
Best observed pass rate per model · AutomationBench 1.0.6
All three settings sit on the frontier, from $0.18 a task.
Each line is one model's effort settings, pass rate against cost per task. Nothing in this comparison is both cheaper and more accurate than any Hydrogen configuration. Gemini 4 Carbon still sets the highest score, 2.8 points above Hydrogen's best, at $1.94 a task.
| Model | Low | Medium | High |
|---|---|---|---|
| Gemini 4 Hydrogen | 30.6%$0.18 | 46.7%$0.41 | 51.1%$0.82 |
| Gemini 4 Carbon | — | — | 53.9%$1.94 |
| Gemini 4 Argon | — | — | 51.3%$1.63 |
API cost per task, list-price estimate, with caching · pass rate at each effort, low to max · Cost comparisons use the models in this run's metrics export. Claude Fable 5.2 and Claude Sonnet 5.5 are pass-rate results from the zapier.com/benchmarks leaderboard, with Anthropic's default fallbacks.
Operations is the strongest domain, and it peaks at medium.
Each line runs across the three effort settings. Operations reaches 65.0% at medium and gives back a point at high. HR is the hardest: it gains 9.2 points across the full range, the narrowest of any domain, and finishes at 34.2%.
| Domain | Low | Medium | High |
|---|---|---|---|
| Sales | 15.4%18/117 | 47.0%55/117 | 52.1%61/117 |
| Marketing | 41.0%41/100 | 50.0%50/100 | 57.0%57/100 |
| Operations | 50.0%50/100 | 65.0%65/100 | 64.0%64/100 |
| Support | 18.0%18/100 | 36.0%36/100 | 42.0%42/100 |
| Finance | 36.7%44/120 | 55.0%66/120 | 59.2%71/120 |
| HR | 25.0%30/120 | 29.2%35/120 | 34.2%41/120 |
Pass rate · passed / tasks per cell · darker = higher
$0.82 a task at high, $0.41 at medium.
Low has the lowest spend per passed task, $0.58, against $0.88 at medium and $1.61 at high. Spend per pass counts the cost of every failed task too. These are estimates at Google's stated rates, not billed costs.
| Effort | Cost / task | Passed task | Failed task | Spend per pass |
|---|---|---|---|---|
| Low | $0.18 | $0.19 | $0.17 | $0.58 |
| Medium | $0.41 | $0.42 | $0.40 | $0.88 |
| High | $0.82 | $0.76 | $0.88 | $1.61 |
USD, list-price estimate, with caching · spend per pass = all task costs ÷ passes, failures included
No task hits the step cap: 0 of 657, at every effort.
Tool calls climb from 53.4 per task at low to 116.4 at high, and steps from 10.6 to 18.6, well inside the 50-step cap. Reasoning tokens grow 8.7× across the same range, from 5,001 to 43,666 per task, while the cached share of input rises from 80.4% to 90.0%.
| Effort | Tool calls | Steps | Calls / step | Input tokens | Cached share | At step cap |
|---|---|---|---|---|---|---|
| Low | 53.4 | 10.6 | 5.0 | 215.2k | 80.4% | 0 |
| Medium | 79.7 | 13.5 | 5.9 | 611.8k | 86.3% | 0 |
| High | 116.4 | 18.6 | 6.3 | 1383.4k | 90.0% | 0 |
Mean per task, all tasks included · step cap 50
AutomationBench tests AI agents on end-to-end workflow execution using 47 simulated apps across six business functions: Sales, Marketing, Operations, Support, Finance and HR. Each task boots a small simulated company, hands the agent one request a real operations person would get, then grades the world it leaves behind with deterministic assertions.
CRM records, inbox threads, spreadsheets and support cases, including traps like stale rows and near-duplicate names.
The agent gets one trigger message and tools to search, read, update and send. It can't ask clarifying questions, and there's no human in the loop.
Deterministic assertions check what changed. The task passes only if every one holds.
CostEstimates at Google's stated rates for Gemini 4 Hydrogen: Gemini 3.8 Flash list rates on the Standard paid tier, without the 50% promotion that runs through December 31, 2026. $1.50 input, $0.15 cache read and $7.50 output per 1M tokens, with reasoning billed as output. Gemini implicit caching records no cache writes. These are not billed costs.
ComparisonsCost comparisons use the models in this run's metrics export. Claude Fable 5.2 and Claude Sonnet 5.5 are pass-rate results from the zapier.com/benchmarks leaderboard, with Anthropic's default fallbacks.