Real context moved 6.0pp on Method/Execution Split.
Picks the right method, then does the arithmetic wrong.
Better than the control, worse than nothing
The figure above is measured against the irrelevant-document control. Against no context at all this card is worse: 0% bare against 6% with the best supplied source. What the source repaired is damage the presence of a document did.
~intervals overlap — not separated from no change
The model chooses a correct method but executes it wrongly (or vice versa) — a right-approach/wrong-answer or wrong-approach/right-answer split. The rule
A shop sells 17 boxes holding 24 items each, then 8 items are returned. How many items were sold?
To find the number of items sold, we need to multiply the number of boxes sold by the number of items in each box, then subtract the number of items returned. Number of boxes sold = 17 Number of items in each box = 24 Total items sold = 17 x 24 = 408 Items returned = 8 Items sold = 408 - 8 = 400 Answer: 400
detector fmi_4_1_v1: answer matches oracle
To find the total number of items sold, we first need to find the total number of items sold initially. Total items initially = Number of boxes * Number of items per box Total items initially = 17 * 24 Total items initially = 408 Then, we subtract the number of items returned from the total number of items initially sold. Items sold = Total items initially - Items returned Items sold = 408 - 8 Items sold = 400 Answer: 400
detector fmi_4_1_v1: answer matches oracle
loop 2026-09-09 · meta-llama/llama-3.1-8b-instruct as served · open weights · released 2024-07 · 8B · set s1.4-live · cases v1.7 · conditions v1.0 · ground truth as of 2026-08-09
A shop sells 17 boxes holding 24 items each, then 8 items are returned. How many items were sold?
expected as of 2026-08-09
400. Order matters: multiply before subtracting.
Nothing supplied. The baseline.
To find the number of items sold, we need to multiply the number of boxes sold by the number of items in each box, then subtract the number of items returned. Number of boxes sold = 17 Number of items in each box = 24 Total items sold = 17 x 24 = 408 Items returned = 8 Items sold = 408 - 8 = 400 Answer: 400
fmi_4_1_v1
detector fmi_4_1_v1: answer matches oracle
0–40% · whisker = 95% interval
never pooled · a trap ratio would skew the delta
Supplying context did not reduce this failure, it increased it: 0% with no context against 6% under the strongest supplied source. What survives is therefore not a remainder but a substitution — the failures under context are 3 probes the detector still rejected, dominated by: answer 63.83 ≠ oracle 66.5.
- tool-assisted computation
- verify final against method
fmi_4_1_v1 · 5 repeats per case · best context curated context · rollups + residuals committed · probe log outside git · manifest hash ties them · method