what 83 failed probes cost, relative to a healthy run on the same mode. That gap is the diagnosis.
Measured on the survivors
A provider ate 42% of our probes. The resulting rate looked completely unremarkable.
One model in our panel returned errors on 83 of 200 probes for a single failure mode. The harness discarded the whole loop.
Had it written the manifest, the rate would have looked entirely unremarkable.
Expectation versus reality
You expect a broken run to look broken — timeouts, a wide interval, something obviously wrong on the page.
What you get is a clean number computed on whatever came back. And what came back is not a random sample.
Diagnosis
Those 83 errors cost under 1% of what the dearest healthy model spent on the same mode, and about 6% of the cheapest.
Why: Cheap failures failed before the model ran — rejection. Expensive failures fail after it started — truncation. Two different problems, and the invoice separates them faster than any log.
The arms lost different numbers of probes, which means the comparison between them is between two differently-filtered sets.
Why: This is why denominator drift is a publish-blocking check rather than a warning. A caveat on a biased rate is still a biased rate, and the caveat does not travel with the number.
What we do now
A loop that loses too much is discarded rather than published with an asterisk. We threw away three during the last census and re-ran them — two recovered at lower concurrency, one needed the provider to come back.
Cost per error is now the first thing I check. It is a free diagnostic that nobody thinks of as one.
The number we nearly published was measured on the easy remainder.
Does your eval pipeline record how many calls failed — or only the ones that came back?
Every figure here describes something measured and committed. See the measurements · read the method