Picks the right method, then does the arithmetic wrong.
~intervals overlap — not separated from no change
The model chooses a correct method but executes it wrongly (or vice versa) — a right-approach/wrong-answer or wrong-approach/right-answer split. The rule
A shop sells 17 boxes holding 24 items each, then 8 items are returned. How many items were sold?
No contextPASS
17 × 24 = 408
408 - 8 = 400
Answer: 400
detector fmi_4_1_v1: answer matches oracle
General contextPASS
17 × 24 = 408
408 - 8 = 400
Answer: 400
detector fmi_4_1_v1: answer matches oracle
loop 2026-09-09 · anthropic/claude-sonnet-4.6 as served · closed weights · released 2026-02 · set s1.4-live · cases v1.7 · conditions v1.0 · ground truth as of 2026-08-09
From no context to real context
5 repeats per case
whiskers are 95% intervals · each step is measured from the stop before it, not from zero
Try the other contexts
stored · no model call
1Questionheld fixed
A shop sells 17 boxes holding 24 items each, then 8 items are returned. How many items were sold?
expected as of 2026-08-09
400. Order matters: multiply before subtracting.
2Contextchange me
Nothing supplied. The baseline.
3ResponseNo context
PASS
17 × 24 = 408
408 - 8 = 400
Answer: 400
fmi_4_1_v1
detector fmi_4_1_v1: answer matches oracle
The scoring
Failure rate by context
No context0%[0%–10%] n=50 · eff 35
Irrelevant context0%[0%–10%] n=50 · eff 35
General context0%[0%–10%] n=50 · eff 35
Curated context0%[0%–10%] n=50 · eff 35
0–20% · whisker = 95% interval
Trap vs answerable
No contexttrap 0%answerable 0%
Irrelevant contexttrap 0%answerable 0%
General contexttrap 0%answerable 0%
Curated contexttrap 0%answerable 0%
never pooled · a trap ratio would skew the delta
Residual — what survives
Unchanged by context: 0% with none, 0% with the strongest supplied source. For a control mode that flat line is the result — it is what licenses reading movement elsewhere in this loop as grounding rather than as a document being present. Nothing survives at the best condition on the cases measured here.
Mitigations from the index — the claim
—tool-assisted computation
—verify final against method
fmi_4_1_v1 · 5 repeats per case · best context general context · rollups + residuals committed · probe log outside git · manifest hash ties them · method