Can AI models do real work?

Can AI models do real work?

Zapier Benchmarks measure execution: did the work get done correctly in realistic systems?

Zapier Benchmarks measure execution: did the work get done correctly in realistic systems?

AutomationBench, our lead eval, tests AI agents on end-to-end workflow execution across six domains (Sales, Marketing, Operations, Support, Finance, and HR). It's built on real patterns from 2B+ monthly tasks across 3.7M Zapier customers.

View on GitHub

View on GitHub

View on GitHub

Download the whitepaper

Download the whitepaper

Realistic environments

Each task drops an agent into an isolated environment (CRM records, inbox threads, calendars, and more) with the kind of ambiguity that makes real work hard.

Deterministic scoring

We score the final environment state against fixed success criteria. No LLM-as-judge. No vibes.

Leaderboard

Rank
Model
Score
Cost / task
1
Gemini 3.5 Flash (Medium)
14.5%
$0.87
2
GPT-5.5 (XHigh)
12.9%
$6.31
3
Gemini 3.5 Flash (High)
12.6%
$1.30
4
Gemini 3.5 Flash (Low)
12.2%
$0.65
5
GPT-5.5 (High)
11.3%
$4.06
6
Claude Opus 4.7 (Max)
9.9%
$1.80
7
Gemini 3.1 Pro (High)
9.6%
$0.54
8
GPT-5.5 (Medium)
8.5%
$2.42
9
Claude Opus 4.7 (High)
8.4%
$1.21
10
Claude Opus 4.7 (XHigh)
8.2%
$1.44

By domain

Domain
Top Model
Score
2nd Place model
Sales
GPT-5.5 (XHigh) — OpenAI
17.9%
GPT-5.5 (High) — OpenAI / Gemini 3.5 Flash (High) — Google (tie at 13.7%)
Marketing
GPT-5.5 (XHigh) — OpenAI
20.0%
Claude Opus 4.7 (Max) — Anthropic (18.0%)
Operations
Gemini 3.5 Flash (Medium) — Google
20.0%
GPT-5.5 (High) — OpenAI (17.0%)
Support
GPT-5.4 / Gemini 3.5 Flash (tie) — OpenAI / Google
10.0%
GPT-5.4 (XHigh) — OpenAI (tie at 10.0%)
Finance
Gemini 3.5 Flash (High) — Google
11.7%
Gemini 3.5 Flash (Low) — Google (10.8%)
HR
Gemini 3.5 Flash (Medium) — Google
19.2%
Gemini 3.5 Flash (High) — Google (16.7%)

Try the latest models in Zapier

Try the latest models in Zapier

Want to test frontier models on real workflows—today? Use Zapier to run agents across the tools you already use. Zapier Benchmarks results will be published as evaluations complete.

Want to test frontier models on real workflows—today? Use Zapier to run agents across the tools you already use. Zapier Benchmarks results will be published as evaluations complete.

Try the models

Try the models