Your eval budget is wrong by 300×, and you can't see it
20 models, identical work. One cost $8.14. One cost $0.02.
Not running a failure-mode eval feels free. It is not — you pay for it in production, in incidents nobody can name, and in a roadmap driven by whichever failure was loudest last week.
The measured version costs less than most teams spend on a single week of on-call.
What it actually cost us
| model | cost | minutes |
|---|---|---|
| x-ai/grok-4.6 | $8.14 | 188 |
| openai/o4-mini | $4.41 | 116 |
| anthropic/claude-sonnet-4.6 | $3.95 | 17 |
| openai/gpt-4o | $2.35 | 9 |
| google/gemini-2.5-flash | $0.55 | 8 |
| meta-llama/llama-3.1-8b | $0.02 | 17 |
Three columns of value
A full 20-model panel costs about the price of a team lunch. A dry run to price it costs cents.
Why: The expensive thing is not the tokens. It is the engineering time spent arguing about which model to use without evidence.
You learn which failure modes your chosen model actually has, before a user does, with an interval around each rate.
Why: A named, measured failure is a thing you can put a mitigation against and re-measure. An unnamed one is a recurring incident.
You stop rebuilding evaluation every time a model ships. The taxonomy outlives the panel.
Why: At a three-month release cadence, anything whose unit of work is 'benchmark this generation' is permanently behind.
The two objections
"It's too expensive." It is $25. The dry run that prices it accurately is under a dollar. Run that first and you will never be surprised by the invoice.
"We'd have to build a harness." You would not — start with one failure mode, one model, three arms. That is an afternoon and it produces a number you did not have.
Where I'd start
Pick the failure that bit you most recently. Name it. Write ten cases. Run them with no context, with an irrelevant document, and with your real retrieval. You now have a rate and a control, which is more than most production systems have.
What would it cost your team to find out you were wrong about your model six months from now?
Every figure here describes something measured and committed. See the measurements · read the method