ModelCensusopen-source ai reliability harness
Blog2 Aug 2026methodharnesspost-mortem

A provider ate 42% of our probes and the rate looked fine

Dropout doesn't announce itself as a wide interval. It announces itself as a confident number.

One model in our panel returned errors on 83 of 200 probes for a single failure mode. The harness discarded the whole loop.

Had it written the manifest, the rate would have looked entirely unremarkable.

Expectation versus reality

You expect a broken run to look broken — timeouts, a wide interval, something obviously wrong on the page.

What you get is a clean number computed on whatever came back. And what came back is not a random sample.

200 probes attemptedreturned 59%lost 42%
Timeouts fall on the longest generations. The longest generations are the hard cases. What survives is a biased sample of your own test set.

Diagnosis

The cost told us before the logs did

Those 83 errors cost $0.0119 in total. Healthy models spent $0.19 to $1.54 on the same mode.

Why: Cheap failures failed before the model ran — rejection. Expensive failures fail after it started — truncation. Two different problems, and the invoice separates them faster than any log.

Dropout is never uniform across arms

The arms lost different numbers of probes, which means the comparison between them is between two differently-filtered sets.

Why: This is why denominator drift is a publish-blocking check rather than a warning. A caveat on a biased rate is still a biased rate, and the caveat does not travel with the number.

What we do now

A loop that loses too much is discarded rather than published with an asterisk. We threw away three during the last census and re-ran them — two recovered at lower concurrency, one needed the provider to come back.

Cost per error is now the first thing I check. It is a free diagnostic that nobody thinks of as one.

The number we nearly published was measured on the easy remainder.

Does your eval pipeline record how many calls failed — or only the ones that came back?

Every figure here describes something measured and committed. See the measurements · read the method