Can AI models do real work?

Zapier Benchmarks measure execution: did the work get done correctly in realistic systems?

AutomationBench, our lead eval, tests AI agents on end-to-end workflow execution across six domains (Sales, Marketing, Operations, Support, Finance, and HR). It's built on real patterns from 2B+ monthly tasks across 3.7M Zapier customers.

View on GitHub

Download the whitepaper

Realistic environments

Each task drops an agent into an isolated environment (CRM records, inbox threads, calendars, and more) with the kind of ambiguity that makes real work hard.

Deterministic scoring

We score the final environment state against fixed success criteria. No LLM-as-judge. No vibes.

Leaderboard

Rank

Mannequin

Score

Cost / task

Claude Opus 4.8 (XHigh)

15.5%

$2.36

Claude Opus 4.8 (Max)

15.4%

$3.14

Gemini 3.5 Flash (Medium)

14.5%

$0.87

GPT-5.5 (XHigh)

12.9%

$6.31

Gemini 3.5 Flash (High)

12.6%

$1.30

Gemini 3.5 Flash (Low)

12.2%

$0.65

Claude Opus 4.8 (High)

11.6%

$1.87

GPT-5.5 (High)

11.3%

$4.06

Claude Opus 4.8 (Medium)

11.1%

$1.69

Claude Opus 4.8 (Low)

9.9%

$1.13

By domain

Domaine

Top Model

Score

2nd Place model

Ventes

GPT-5.5 (XHigh) — OpenAI

17.9%

Claude Opus 4.8 (Max) — Anthropic (17.1%)

Marketing

GPT-5.5 (XHigh) — OpenAI

20.0%

Claude Opus 4.7 (Max) — Anthropic (18.0%)

Opérations

Gemini 3.5 Flash (Medium) — Google

20.0%

Claude Opus 4.8 (Max) — Anthropic (19.0%)

Assistance

Claude Opus 4.8 (XHigh) — Anthropic

15.0%

Claude Opus 4.8 (Max/High) — Anthropic (tie at 13.0%)

Finances

Claude Opus 4.8 (Max) — Anthropic

12.5%

Gemini 3.5 Flash (High) — Google / Claude Opus 4.8 (XHigh) — Anthropic (tie at 11.7%)

Claude Opus 4.8 (XHigh) — Anthropic

20.0%

Gemini 3.5 Flash (Medium) — Google (19.2%)

Try the latest models in Zapier

Want to test frontier models on real workflows—today? Use Zapier to run agents across the tools you already use. Zapier Benchmarks results will be published as evaluations complete.

Try the models