We ran the same agent, on the same model, against the same tasks, with and without a compiled domain model. Most of what the baseline spent wasn't doing the work. It was figuring out how.
The bottom line: 90% fewer tokens, more correct#
Same agent, same model, same tasks. The only thing that changed was how it found its way around the APIs: once through a discovery tool, once through Sioma's MCP. Cost, speed, and accuracy all moved together, and that is the part worth explaining.
| Model | Tokens | Wall-clock | Correct (of 77) |
|---|---|---|---|
| gemini-3-1-flash-lite | 1.71M → 167K (−90%) | 144s → 60s (2.4×) | 39 → 53 (51% → 69%) |
| gemini-2-5-flash-lite | 1.06M → 246K (−77%) | 145s → 134s (≈1×) | 21 → 46 (27% → 60%) |
Baseline = the agent handed an API/tool-discovery tool. Sioma = same agent using Sioma MCP.
One row deserves calling out rather than glossing over. On the older model the tokens fell 77% and the clock barely moved. Fewer tokens is not automatically less time: on gemini-3-1-flash-lite the savings came back as 2.4× faster answers, and on gemini-2-5-flash-lite they came back almost entirely as cost and correctness instead. Both rows are from the same runs, which is why both are in the table.
Benchmark setup#
We stood up a benchmark harness mimicking an enterprise-shaped environment: a broad set of APIs, and a gold set of tasks with ground-truth answers, including trick questions designed to punish guessing. Then we ran the comparison twice: once as a baseline agent handed a search entrypoint to find and invoke APIs, and once as the same agent resolving through Sioma's context engine over MCP. The figures above are gemini-3-1-flash-lite.
The baseline was not a straw man. It could search the available APIs by name and description, read a full schema on demand for anything it found, and retry after a failed call. That is the setup teams ship today: hand the agent a discovery tool and let it work the rest out at runtime.
Where the tokens went: tool selection, API exploration#
The interesting result isn't the total, it's what the baseline was spending on. The agent, discovering which tools and APIs it needed, burned most of its token budget: reading schemas, choosing among similar endpoints, planning the order, retrying. Sioma MCP served steps to execute. The 90% of tokens the baseline spent and Sioma didn't weren't doing the task. They were the agent re-deriving the structure of the systems.
That is the cost of an agent figuring out how your systems fit together. It does it probabilistically, on every task: the right endpoints, their order, their constraints. It is building the prerequisite to the work, before the work starts.
Why better prompting doesn't remove it#
You can shrink the tax with tighter tool descriptions, curated toolsets, skills, and few-shot plans. All of it helps. But as long as the agent does this work at runtime, it costs something every time, and because the process is probabilistic, some of those attempts will be wrong.
Sioma resolves intent against a compiled model instead. Resolution is deterministic, and it can return "no match" rather than guess.
The idea isn't new#
You have probably heard of MCP tool overload, where loading more tool servers into an agent's context makes it slower and worse at choosing the right one. This benchmark is that phenomenon at enterprise scale: hundreds of endpoints rather than dozens, measured on an enterprise-shaped surface.
What compiling changes#
Sioma moves the derivation to build time. It compiles a domain model of the business, and at runtime intents resolve against that model into exact steps. At runtime there is usually a trade-off between cost, speed, and correctness, because exploration is their common cause. Sioma removes the exploration, and the cost moves to build time, where you pay it once. The 90% cut, the 2.4×, and the jump from 51% to 69% aren't three separate optimizations. They are one thing: the removal of exploration.
Run it yourself#
Sioma is in closed beta, and the benchmark runs against your own system. Set up your APIs, choose the model, bring your agent, connect Sioma MCP, and generate your gold set. Then run it both ways and compare. Sign up to run the benchmark.