Real context moved 0.0pp on Paraphrase Non-Invariance.
~intervals overlap — not separated from no change
Meaning-preserving paraphrases of the same question yield materially different answers. The rule
How many bones are in the adult human body?
An average adult human skeleton has 206 bones in total. This is typically divided into: • Axial skeleton: 80 bones (skull, vertebral column, ribs, sternum) • Appendicular skeleton: 126 bones (limbs and their girdles)
detector fmi_5_2_v1: answers invariant across paraphrases
An average adult human skeleton is made up of 206 bones.
detector fmi_5_2_v1: answers invariant across paraphrases
loop 2026-08-15 · openai/o4-mini as served · closed weights · released 2025-04 · reasoning model · set s1.3-live · cases v1.6 · conditions v1.0 · ground truth as of 2026-08-09
How many bones are in the adult human body?
expected as of 2026-08-09
206. 206 in a typical adult. Widely misremembered, which is what makes the phrasings diverge.
Nothing supplied. The baseline.
An average adult human skeleton has 206 bones in total. This is typically divided into: • Axial skeleton: 80 bones (skull, vertebral column, ribs, sternum) • Appendicular skeleton: 126 bones (limbs and their girdles)
fmi_5_2_v1
detector fmi_5_2_v1: answers invariant across paraphrases
0–40% · whisker = 95% interval
never pooled · a trap ratio would skew the delta
Supplying context did not reduce this failure, it increased it: 0% with no context against 7% under the strongest supplied source. What survives is therefore not a remainder but a substitution — the failures under context are 0 probes the detector still rejected. Nothing survives at the best condition on the cases measured here.
- paraphrase-augmented training
- canonicalization
fmi_5_2_v1 · 5 repeats per case · best context curated context · rollups + residuals committed · probe log outside git · manifest hash ties them · method