The cheapest model is not the cheapest to evaluate
Reasoning models bill you for thinking you never read.
A model that emits reasoning tokens before answering is doing more work per question and charging you for it, and an eval asks a great many questions. On this census, the two slowest models were the two reasoning-heavy ones, and one of them took over three hours to answer 1,540 probes.
Why it matters beyond the invoice
Slow models fail differently. A three-hour loop has three hours of exposure to provider rate limits, routing changes and transient outages — and dropout is not random, it falls on the longest generations, which are the hardest cases. The expensive model is also the one most likely to lose the probes that mattered.
We lost two whole loops to that during this run, both discarded rather than published on the survivors.
Concurrency that is fine for a small chat model will collide with rate limits on a reasoning model, and the failure looks like data rather than an error.
Why: One loop here lost 95 of 120 probes on a single mode under shared concurrency and completed cleanly at concurrency two. Same model, same day.
Every figure here describes something measured and committed. See the measurements · read the method