A 0% score that meant two opposite things
Perfect calibration and refusing everything were indistinguishable for months.
Post-Cutoff Fabrication scores by hedging: the model should flag that a volatile fact may have changed rather than state it flatly. Every case in our set was one of those.
Which made a score of 0% completely unreadable.
Expectation versus reality
We read a 0% as well-calibrated behaviour. A model that hedges appropriately scores zero.
So does a model that answers "I can't be certain about anything" to every question on earth. Nothing on the card separated them, and one of those is a product nobody wants.
Diagnosis
Every question was one where caution was correct. There was no question where caution was the failure.
Why: A mode with only one failure direction cannot distinguish the behaviour you want from its degenerate extreme. The set has to be able to punish both.
Our own share-card rule rejected every claim on this mode under NO_CONTROL_TAG — 84 of 846 candidates, all one mode.
Why: The gate was written to stop bad cards travelling. It ended up being the thing that told us the mode was broken.
The fix, and what it cost
Six settled facts, fixed long before the earliest training cutoff on the panel and unchanged since. On those, declining to answer is the failure. The detector branches on the shape of the oracle rather than the tag.
Cost: a full 20-model re-run. $25 and thirteen hours. First result out of it showed something the old set structurally could not — under an irrelevant document, one model started declining questions it could answer perfectly well.
The old set would have rendered that as a clean 0% and called it good calibration.
If your metric can only be failed in one direction, it is measuring half a behaviour.
Which of your metrics would a system score perfectly on by simply refusing to act?
Every figure here describes something measured and committed. See the measurements · read the method