Today we're releasing AutomationBench, an open benchmark that measures whether AI models can complete real business workflows.
Current model evaluations tell you whether an LLM can answer math olympiad questions, write code, or reason through a logic puzzle. Those are important, but they don't tell you the thing that matters most to enterprises: whether it can get work done. As in, find the right CRM records, send the right follow-up, update the right systems, and get to a verifiable end state without breaking anything along the way.
AutomationBench fills that gap, evaluating AI agents on end-to-end business execution across the tools enterprises actually use. It scores models on proof of outcome: did the work get done correctly, or didn't it?
It's a benchmark for outcomes, not output.
How it works
AutomationBench evaluates models across six business domains (Sales, Marketing, Operations, Support, Finance, and HR), selected based on the most common use-case patterns across the 3.7M companies and 2B monthly tasks Zapier sees.
Each task drops an AI agent into a realistic environment: a CRM with live data, an inbox with threads, a calendar with conflicts. The agent gets a starting prompt (e.g., a user request or a Slack message) and has to figure out what to do. The environments include the kind of ambiguity that makes real work hard: similarly named contacts, inconsistent formats, multi-step tool chains where a wrong call cascades.
Scoring is deterministic. We check the final state of the environment against a set of success criteria. Either the right records were updated and the right messages were sent, or they weren't. There's no LLM-as-judge or otherwise subjective grading.
The evaluation uses a public/private split, so model providers can receive validated results without training on the full evaluation set. We'll also publish cost-versus-performance comparisons as we evaluate more models.
Why Zapier
Workflow success is the missing yardstick for enterprise AI, and building a credible benchmark requires two things: real workflow patterns and real tool complexity. Zapier has both at a scale in a way no one else does.
Our platform processes over 2 billion AI tasks per month across 3.7 million total companies. Our developer ecosystem has 9,000+ native app integrations with 66,000+ triggers and actions, plus 140,000+ private integrations built by enterprises for their own internal systems. That gives us uniquely deep visibility into what business automation actually looks like in practice: messy, cross-system, cross-department, multi-step.
We initially built AutomationBench for internal use, to help evaluate which models to deploy across Zapier, and existing benchmarks didn't measure the element we cared about (i.e., can this model actually do the work?). It proved useful enough that we're now releasing it publicly so model providers and enterprises can use it too.
Getting started
AutomationBench launches today with a public task set and supporting materials. The full leaderboard, methodology, and results are available at zapier.com/benchmarks.
Verified private-set evaluation is available to model providers by request. We're also working with frontier AI labs and enterprise partners on early evaluation as we expand to additional domains.
