Price per token tells you what a call costs. It tells you nothing about how many calls it takes to trust the answer.
the eval cost inversion
Cheap per token, expensive to measure
Everyone budgets an eval from the price sheet. Ours was wrong by an order of magnitude, and the invoice was the smaller half.
Everyone budgets an eval from the price sheet. Price per million tokens, times prompts, done.
That number was wrong for us by an order of magnitude. And the money was the smaller problem.
Why the price sheet accumulates debt
Reasoning models bill you for thinking you never read. Every model on our panel answered the same 1,540 probes. The slowest took 188 minutes and a third of the entire census bill on its own. gpt-4o took nine minutes and under a tenth.
Wall clock is the line item nobody prices. A three-hour loop is three hours of exposure to rate limits, routing changes and provider outages.
The reality on the ground
Here is the part that should worry you. Dropout is not random. It falls on the longest generations, and the longest generations are the hardest cases.
So the expensive model is also the one most likely to quietly lose the probes that mattered. We lost two whole loops that way. Both were discarded, not published on whatever survived.
Estimate minutes per model alongside dollars. A loop that runs overnight has a failure surface a lunchtime loop does not.
Concurrency that is fine for a small chat model collides with rate limits on a reasoning model.
Why: And the collision does not look like an error. It looks like data.
It costs cents and tells you the real shape of the bill, including the thinking tokens.
When the price sheet is right
For small, non-reasoning models answering short prompts, token price and eval cost track closely. gemini-2.5-flash ran the full set in eight minutes for about 2% of the census bill. The price sheet only starts lying once thinking tokens and long wall clocks enter.
The expensive model is also the one most likely to lose the hard cases. Budget for the re-run, not just the run.
When you budgeted your last eval, did anyone price the reasoning tokens — or the re-runs?
Every figure here describes something measured and committed. See the measurements · read the method