cost spread across 20 models doing identical work. One was a third of the bill.
The 338× question
Twenty models, identical work, 338x spread. Your eval budget is wrong and you cannot see it.
Not running a failure-mode eval feels free. It is not — you pay for it in production, in incidents nobody can name, and in a roadmap driven by whichever failure was loudest last week.
The measured version costs less than most teams spend on a single week of on-call.
Where the cost actually went
| model | share of census cost | minutes |
|---|---|---|
| x-ai/grok-4.6 | 32% | 188 |
| openai/o4-mini | 17% | 116 |
| anthropic/claude-sonnet-4.6 | 16% | 17 |
| openai/gpt-4o | 9% | 9 |
| google/gemini-2.5-flash | 2% | 8 |
| meta-llama/llama-3.1-8b | < 0.1% | 17 |
Three columns of value
A full 20-model panel costs less than a single week of on-call. A dry run to price it costs under 1% of the full run.
Why: The expensive thing is not the tokens. It is the engineering time spent arguing about which model to use without evidence.
You learn which failure modes your chosen model actually has, before a user does, with an interval around each rate.
Why: A named, measured failure is a thing you can put a mitigation against and re-measure. An unnamed one is a recurring incident.
You stop rebuilding evaluation every time a model ships. The taxonomy outlives the panel.
Why: At a three-month release cadence, anything whose unit of work is 'benchmark this generation' is permanently behind.
The two objections
"It's too expensive." It is not — the whole census cost less than a week of on-call, and the dry run that prices it accurately is under 1% of the full run. Run that first and you will never be surprised by the invoice.
"We'd have to build a harness." You would not — start with one failure mode, one model, three arms. That is an afternoon and it produces a number you did not have.
Where I'd start
Pick the failure that bit you most recently. Name it. Write ten cases. Run them with no context, with an irrelevant document, and with your real retrieval. You now have a rate and a control, which is more than most production systems have.
What would it cost your team to find out you were wrong about your model six months from now?
Every figure here describes something measured and committed. See the measurements · read the method