a run sat waiting on dead sockets after the machine slept. No error, no output, and a process that looked alive.
The harness threw the loop away
13.3 hours, 30,800 probes, three discarded loops — and none of it was the models' fault.
Twenty models. 30,800 probes. 13.3 hours of run time. The number that surprised me most was how little the failures cost: under 1% of a healthy run.
Expectation versus reality
I expected the failures in a twenty-model census to come from the models. Three loops were discarded and re-run. None of the three was a model problem.
Diagnosis
Nine failing modes, combined, cost less than 1% of what the dearest healthy model spent on that one mode — and about 6% of the cheapest.
Why: A call that dies mid-generation still bills for the tokens it produced. A call refused at the gate bills for almost nothing. The provider was rejecting requests before the model ever ran.
The harness refused to write rates from whatever came back.
Why: That is the whole value of a dropout gate. The rates from the surviving probes would have looked completely reasonable.
Node's fetch has no default timeout. After the machine suspended, in-flight requests were dead and the run waited on them forever.
Why: A request with no deadline is not a slow request. It is a hung process that reports as running.
What we run now
- An explicit timeout on every outbound call
- Cost per error recorded on every loop
- A loop with unequal dropout between arms is discarded, never salvaged — dropout is not random, and it lands on the hard cases
A defect in an eval does not crash. It produces a plausible number in the wrong direction.
What does your eval pipeline do when a provider fails half the calls — stop, or publish the half that came back?
Every figure here describes something measured and committed. See the measurements · read the method