Can AI models do real work? Zapier's AI benchmark assessment measures execution in realistic systems.
AutomationBench tests AI agents on end-to-end workflow execution using 47 real tools across six business functions: Sales, Marketing, Operations, Support, Finance, and HR. It's built on real patterns from 2B+ monthly tasks across 3.7M companies.
Realistic environments
Each task drops an agent into an isolated environment (CRM records, inbox threads, calendars, and more) with the kind of ambiguity that makes real work hard.
Deterministic scoring
We score the final environment state against fixed success criteria. No LLM-as-judge. No vibes.
How AutomationBench works
Each task boots a tiny simulated company, hands the agent a request a real ops person would get, and grades the world it leaves behind. The agent's reply isn't scored — the data is.
600+
Held-out evaluation tasks
6
Business domains
47
Simulated apps
~500
API endpoints
Boot a tiny company
The task seeds a fresh simulated business: CRM records, inbox threads, spreadsheets, support cases. Every model starts from this exact same world — including its traps: stale rows, near-duplicate names, policies buried in an inbox.
The agent works alone
One trigger message kicks things off. The agent gets tools to search, read, update, and send — no clarifying questions, no human in the loop. It has to find what it needs the way a new hire would: by looking.
Grade the end state
When the agent stops, deterministic assertions inspect the world it left behind. Was the record updated? Did the right team get the email — and the wrong teams not? The outcome happened or it didn't.
One task, end to end · sales.multi_hop_lookup
CRM · three Meridians
| Meridian Corp — Platform Deal | €120,000 |
| Meridian Solutions — Platform Deal | $150,000 |
| Meridian Corporation — Platform Deal | $95,000 |
Near-duplicate names. Update the wrong record and the task fails.
Sheets · stale rows
| EUR → USD | 1.10 | Jan 10 |
| EUR → USD | 1.30 | Jan 25 |
| Tier | Mid-Market | Dec 15 |
| Tier | Enterprise | Jan 12 |
Both sheets contain outdated entries. The newest row wins: €120,000 × 1.30 = $156,000, tier = Enterprise.
Inbox · the routing policy
"Enterprise → executive-team@ · Mid-Market → vp-sales@ · SMB → smb-team@ … if the account has any open Critical/High escalations, also notify support-escalation@."
The rules aren't in the prompt — the agent has to find this email. And the open Critical case sits on the parent account, one hop up the hierarchy.
The grade · six assertions on the final state
- Opportunity
Meridian Corp — Platform Dealstage isClosed Wonthe right record, not a lookalike - Email sent to
executive-team@naming the deal,$156,000, andEnterprisecorrect tier from the newest row, correct FX math - Email sent to
support-escalation@with the same detailsfound the parent account's open Critical case - No win notice sent to
vp-sales@the stale tier would have routed here - No win notice sent to
smb-team@a lookalike account's tier - No win notice sent to
sales-team@the lazy default for "everything else"
Official metric
task_completed_correctly
Strict pass/fail. Every scored assertion must pass. Leaderboard scores run against a held-out private evaluation set; the public 600-task set is for research and experimentation. Run-to-run variance is typically within 1%.
Diagnostic only
partial_credit
Fraction of assertions passed. Useful for debugging and training signal — and as a dense reward when the benchmark is used as an RL environment. Not part of the headline score.
Frequently asked
Why are the scores so low?
The benchmark measures full workflow completion across multiple apps, not isolated reasoning. The most striking failure mode: models declare success while actually failing. In one analysis, 72% of Opus's failures, 91% of Gemini's, and 84% of GPT-5.4's involved this false confidence — the agent reports the task done, but the world state is wrong.
Other common failure modes: not persisting when first searches don't find data, assuming data lives where it intuitively "should" (CRM instead of Sheets), processing some items from a list then summarizing as if done, paraphrasing instructions instead of following them exactly.
A task is only complete when business state is correct end-to-end. Getting most of the way there still fails strict scoring.
How does the agent actually interact with the apps?
Two tools: search runs BM25 keyword search over API schemas and returns the top 5 candidates; execute mimics a curl/fetch with method, URL, and body. Discovering the right endpoints is part of the challenge.
Leaderboard scores run in API mode. Behind the simulated apps, Pydantic models are the source of truth — schemas, pagination, required fields, and 4xx error cases behave like the real APIs, but state lives locally so runs are reproducible. Agents get up to 50 steps per task (rarely hit). AutomationBench also supports Zapier-tool mode and a Limited Zapier toolset for experiments on how tool surface shape affects performance.
How is this different from other agent benchmarks?
Many AI benchmarks test coding, browsing, QA, or general reasoning. AutomationBench focuses on whether agents can complete cross-app business work reliably across sales, marketing, ops, support, finance, and HR.
Key design choices: deterministic final-state assertions instead of LLM-as-judge, multi-app workflows instead of single-turn Q&A, simulated tools and APIs instead of screenshot-based browsing, and both positive and negative assertions — the negatives exist to prevent shotgun reward hacking, like emailing everyone in the company instead of the specified recipients.