Today, AutomationBench joins Fireworks' Specialized Intelligence Index (SII), bringing business workflow automation to a growing collection of practitioner-built benchmarks that compare AI models on real work. We're one of 32 new partners in this round, alongside teams covering fields like finance, healthcare, software, and ML research.
AutomationBench measures whether an AI agent can finish a business workflow end to end across sales, marketing, operations, support, finance, and HR. Being part of the SII gives teams a clearer way to see which models they can trust to run those workflows, and how the same models compare on other kinds of specialized work.
The results back up something we see every day on Zapier: a model that does well on general benchmarks won't necessarily handle real automation work.
Benchmarks need to reflect how work actually gets done
Most AI benchmarks test whether a model can answer a question, solve a puzzle, or write code. A model can score well on those and still struggle with an ordinary operations request. To get a workflow right, an agent has to pick the right record out of several with similar names, follow a rule that lives in an email thread instead of in its instructions, and update several apps without touching the wrong one.
The tricky part is that a wrong step rarely causes an error. The workflow carries on, and nobody notices until a customer or a colleague does. That's why AutomationBench checks the outcome of each task instead of the agent's answer.
How AutomationBench tests real business work
We built AutomationBench on the patterns we see across more than 2 billion monthly tasks from 3.7 million companies on Zapier. Each task sets up a small simulated company with a CRM, an inbox, spreadsheets, and support cases, spread across 47 simulated apps. The agent gets one request, the kind an operations hire might get on Slack, and has to work out the rest on its own without asking any clarifying questions.
Here's one task. The agent is told a deal with Meridian Corp has closed, and it needs to mark the deal as won and let the right team know. The CRM has three accounts with nearly identical names. The spreadsheet that lists each account's tier has an old row and a newer one, and only the newer one is right. The rule for which team hears about which deals is buried in an email in the inbox. To pass, the agent has to update the right deal, email the teams that rule points to, and email nobody else.
When the agent finishes, we check what actually changed in each app instead of what the agent says it did. A task only passes if every check holds, because in a live business, mostly right is still wrong. The leaderboard runs on 657 of these tasks, and we keep them private so models can't be trained on the answers.
How models perform on AutomationBench
The SII shows the top 15 models from our leaderboard, each at its best-scoring setting. Score is the share of tasks a model completed with every check passing.
Even the best model fails about half the time, and most of those failures are hard to spot. Gemini 4 Argon leads at 51.3%, with Claude Sonnet 5.5 second at 44.7%. "The failure mode that surprised us most is false confidence: the agent reports success while the data says otherwise," says Robin Salimans, Principal Engineer, Applied AI at Zapier. "That's the majority of failures we see, and it's the single biggest gap between a good demo and something you'd let run part of a business."
Price doesn't predict performance. GPT-6 Sol scores about 80% of what GPT-6 Astra does for roughly a sixth of the cost per task, and Claude Sonnet 5.5 beats Claude Opus 5.5 while costing less.
The cheapest models are for simple steps. GPT-6 Luna and DeepSeek V4 Flash cost $0.04 a task but complete fewer than a quarter of full workflows. They're a better fit for single steps like sorting a ticket into a category or tidying up text.
What this means for choosing a model
Match the model to the job. For a single AI step that categorizes a support ticket or pulls details out of an email, a cheaper model will usually be fine, and you can compare which models are available on Zapier by cost tier. For an agent that has to find records across several apps and act on them, it's worth paying for a model near the top of the board.
Then check the result yourself instead of trusting the agent's report. For anything that touches customers or money, add a search step after the AI step to look up the record, and a filter that stops the Zap if the record wasn't updated the way you expected. When a new model comes out, swap it into your AI by Zapier step in one click and run it on a few real examples before switching for good.
Benchmarks should reflect the work people actually need done. By bringing AutomationBench to the SII, we're adding business workflows to a shared place where teams can compare models on the work they care about. Explore the full leaderboard at zapier.com/benchmarks, see how AutomationBench sits alongside the rest of the SII, or read the AutomationBench paper for the full methodology.
