A provider went down mid-run, and the harness threw the loop away
Operational notes from 13.3 hours of census, including the bug that cost six of them.
Twenty loops, 30,800 probes, 13.3 hours. Three loops were discarded and re-run, one bug cost most of a night, and none of it was the models' fault.
The provider outage
One model errored on nine of ten modes — total loss on most of them — at a cumulative cost of $0.0119. That number is the diagnosis: healthy models spent $0.19 to $1.54 on the same mode, so these calls were being refused immediately rather than timing out mid-generation. The provider behind the id was rejecting requests, and it recovered an hour later.
The harness discarded the loop rather than writing rates computed on whatever survived. That is the whole value of the dropout gate: the resulting numbers would have looked entirely reasonable.
The bug that cost a night
The machine suspended overnight. On resume, the in-flight HTTP requests were dead, but Node's fetch has no default timeout — so the run sat waiting on sockets that would never answer. Six and a half hours of no output, no error, and a process that looked alive. Zero open sockets was the tell.
A request with no deadline is not a slow request, it is a hung process that reports as running.
Why: The failure mode is indistinguishable from a long generation until you check the socket table, and by then you have lost the window.
Errors that cost nothing failed before the model ran. Errors that cost something failed after it started. Those are different problems with different fixes.
Why: It is the fastest way to tell a rejected request from a truncated one without reading a single log line.
An evaluation harness is software, software has defects, and a defect in an eval does not crash — it produces a plausible number in the wrong direction.
Every figure here describes something measured and committed. See the measurements · read the method