Real context moved 0.0pp on Method/Execution Split.
~intervals overlap — not separated from no change
The model chooses a correct method but executes it wrongly (or vice versa) — a right-approach/wrong-answer or wrong-approach/right-answer split. The rule
A shop sells 17 boxes holding 24 items each, then 8 items are returned. How many items were sold?
First, calculate the total number of items in the 17 boxes: 17 boxes × 24 items per box = 408 items Then, subtract the 8 items that were returned: 408 − 8 = 400 items Answer: 400
detector fmi_4_1_v1: answer matches oracle
First, calculate the total number of items sold initially: 17 boxes × 24 items per box = 408 items Then, subtract the 8 items that were returned: 408 − 8 = 400 items Answer: 400
detector fmi_4_1_v1: answer matches oracle
loop 2026-08-15 · qwen/qwen3-235b-a22b-2507 as served · open weights · released 2025-07 · 235B-A22B · set s1.3-live · cases v1.6 · conditions v1.0 · ground truth as of 2026-08-09
A shop sells 17 boxes holding 24 items each, then 8 items are returned. How many items were sold?
expected as of 2026-08-09
400. Order matters: multiply before subtracting.
Nothing supplied. The baseline.
First, calculate the total number of items in the 17 boxes: 17 boxes × 24 items per box = 408 items Then, subtract the 8 items that were returned: 408 − 8 = 400 items Answer: 400
fmi_4_1_v1
detector fmi_4_1_v1: answer matches oracle
0–20% · whisker = 95% interval
never pooled · a trap ratio would skew the delta
Unchanged by context: 0% with none, 0% with the strongest supplied source. For a control mode that flat line is the result — it is what licenses reading movement elsewhere in this loop as grounding rather than as a document being present. Nothing survives at the best condition on the cases measured here.
- tool-assisted computation
- verify final against method
fmi_4_1_v1 · 5 repeats per case · best context curated context · rollups + residuals committed · probe log outside git · manifest hash ties them · method