We deleted 1,167 findings in an afternoon
They were the same ten questions, counted fifty times.
We had 1,218 separated model pairs in the corpus. Real differences, computed correctly, ready to publish.
Then we fixed the interval calculation and 1,167 of them vanished.
Expectation versus reality
The assumption: n=50 means fifty observations, so the interval is the one you get from fifty observations. That is what every stats library will hand you if you pass it a numerator and a denominator.
The reality: our n of 50 was ten distinct questions asked five times each. A model that fabricates a citation for a prompt usually fabricates it on all five repeats of that prompt. Those are not five observations. They are one, sampled five times.
Diagnosis
Five repeats of ten cases costs the same as two repeats of twenty-five. We chose the first because it was less writing.
Why: Writing cases is the expensive part — a case needs a prompt, an oracle, and four arms. Repeats are free. So the incentive points directly at the design that buys least.
An earlier version of our methodology page described this as a known approximation with the fix as future work.
Why: They were narrow by a factor of three to four. That is not slight, and describing it as slight is how it stayed unfixed for months.
What we do now
Every interval is computed at the effective sample size — n divided by one plus (repeats minus one) times the intra-case correlation — with the correlation measured per mode rather than assumed.
Both numbers ship on the card: n=50 · eff 15. A reader who only sees the first is being told the wrong thing.
The 1,167 that vanished were never findings. They were a design choice showing up as statistics.
Spend on cases. Repeats are nearly free precision on paper and almost none in fact.
If you report an n on your evals — is it distinct cases, or cases times repeats?
Every figure here describes something measured and committed. See the measurements · read the method