spread in the cost of a single benchmark run across tasks on one public agent leaderboard
Evaluation became the expensive part
$40,000 for one leaderboard run. Costs spanning four orders of magnitude. This is now a budget conversation.
A public agent leaderboard reported $40,000 for 21,730 rollouts across nine models and nine benchmarks. Within that, the cost of a single run varied by four orders of magnitude depending on the task.
Evaluation stopped being a rounding error. Nobody has told the budget.
The status quo tax
Most organisations still treat evaluation as something you do at the end, out of whatever is left. That worked while a model call cost nothing and an eval meant a spreadsheet of fifty prompts.
Agentic evaluation broke that. A rollout is not one call — it is a trajectory, with retries, tool calls and reasoning tokens, and its cost is unpredictable by construction. Budget it from an average and you will be wrong in both directions, badly.
Three columns of value
Knowing which shape of evaluation you are buying is worth more than negotiating the token price. A fixed-probe failure-mode census across twenty models cost us under a tenth of one percent of that leaderboard run; an open-ended agent trajectory suite sits orders of magnitude higher.
Why: The variance is in the design, not the vendor. A dry run that prices your actual shape costs a fraction of a percent of the full run and removes the surprise entirely.
An evaluation you cannot afford to re-run is an anecdote. If the quarterly cost is high enough to need approval, it will be skipped in the quarter you most need it.
Why: The value of a measurement is almost entirely in the second one. A single number tells you nothing about whether anything changed.
Separate the two budgets. Failure-mode measurement is cheap, stable and reusable across models. Agent trajectory evaluation is expensive, volatile and specific to your application.
Why: Funded as one line, the expensive half consumes the cheap half, and the cheap half was the one that transferred to the next model generation.
The two objections
"We'll just use a cheaper model to evaluate." Then you have two systems with unmeasured failure modes and no way to tell which one produced the disagreement. Our nineteen instrumented modes are scored by rule for exactly this reason.
"Our vendor includes evaluation." Ask which failure modes it names and what the interval on each rate is. If the answer is a single quality score, you have bought a number that cannot be wrong, which means it cannot be useful.
Where I would start
Price your actual shape before you design around it. Run ten cases through one model, measure what it cost, and multiply. It is an afternoon, it costs a rounding error, and it is the difference between a budget and a hope.
The expensive part of evaluation is no longer the tokens. It is discovering the bill after you have committed to the design.
What does one full re-run of your evaluation suite cost — and has anyone been asked recently?
Every figure here describes something measured and committed. See the measurements · read the method