Delete your significance threshold
Someone proposed a 5pp trigger on a cell whose interval was 27pp wide. That's a noise detector.
We were handed a content plan with two rules for mining our census for social posts. Flag anything where context moved the rate more than 5 percentage points. Flag anything where the model spread grew more than 15.
Both are noise detectors. Neither should exist.
Why a threshold accumulates debt
At the sample sizes involved, a single cell's confidence interval is around twenty-seven percentage points wide. A five-point trigger sits comfortably inside the noise.
It will fire constantly. It will fire on nothing.
The deeper problem is that a threshold is a dial, and a dial gets turned when the results are disappointing. Not maliciously. Just gradually.
What we use instead
A claim is admissible when two intervals do not overlap, at the resolution the measurement was actually made. One model, one mode, one condition.
There is no parameter. So there is nothing to quietly adjust.
It is a harsh rule. Applied to our own corpus it rejects the overwhelming majority of candidate comparisons — 845 candidates, 28 survive.
When a threshold is fine
Monitoring. If you are watching a production metric for drift and you need an alert, a threshold is the correct tool and interval separation is overkill.
The distinction is whether you are detecting a change or claiming one. Alerts can be wrong cheaply. Published claims cannot.
Interval separation is the threshold. There is nothing else to choose.
What's the threshold in your eval — and who picked the number?
Every figure here describes something measured and committed. See the measurements · read the method