Real context moved 0.0pp on Method/Execution Split.
~intervals overlap — not separated from no change
The model chooses a correct method but executes it wrongly (or vice versa) — a right-approach/wrong-answer or wrong-approach/right-answer split. The rule
A shop sells 17 boxes holding 24 items each, then 8 items are returned. How many items were sold?
The shop initially had 17 boxes with 24 items each. Total items initially = Number of boxes * Items per box Total items initially = 17 * 24 Calculate 17 * 24: 17 * 24 = 17 * (20 + 4) = 17 * 20 + 17 * 4 = 340 + 68 = 408 So, there were 408 items in total. Then, 8 items were returned. This means 8 items were not sold. Items sold = Total items initially - Items returned Items sold = 408 - 8 Calculate 408 - 8: 408 - 8 = 400 Answer: 400
detector fmi_4_1_v1: answer matches oracle
The problem asks for the total number of items sold. First, calculate the total number of items in the boxes: 17 boxes * 24 items/box. Then, subtract the number of returned items from the total number of items initially sold. Total items in boxes = 17 * 24 17 * 24 = 17 * (20 + 4) = 17 * 20 + 17 * 4 = 340 + 68 = 408 Number of items returned = 8 Number of items sold = Total items in boxes - Number of items returned Number of items sold = 408 - 8 = 400 Answer: 400
detector fmi_4_1_v1: answer matches oracle
loop 2026-08-15 · google/gemini-2.5-flash-lite as served · closed weights · released 2025-07 · set s1.3-live · cases v1.6 · conditions v1.0 · ground truth as of 2026-08-09
A shop sells 17 boxes holding 24 items each, then 8 items are returned. How many items were sold?
expected as of 2026-08-09
400. Order matters: multiply before subtracting.
Nothing supplied. The baseline.
The shop initially had 17 boxes with 24 items each. Total items initially = Number of boxes * Items per box Total items initially = 17 * 24 Calculate 17 * 24: 17 * 24 = 17 * (20 + 4) = 17 * 20 + 17 * 4 = 340 + 68 = 408 So, there were 408 items in total. Then, 8 items were returned. This means 8 items were not sold. Items sold = Total items initially - Items returned Items sold = 408 - 8 Calculate 408 - 8: 408 - 8 = 400 Answer: 400
fmi_4_1_v1
detector fmi_4_1_v1: answer matches oracle
0–20% · whisker = 95% interval
never pooled · a trap ratio would skew the delta
Unchanged by context: 0% with none, 0% with the strongest supplied source. For a control mode that flat line is the result — it is what licenses reading movement elsewhere in this loop as grounding rather than as a document being present. Nothing survives at the best condition on the cases measured here.
- tool-assisted computation
- verify final against method
fmi_4_1_v1 · 5 repeats per case · best context general context · rollups + residuals committed · probe log outside git · manifest hash ties them · method