Method/Execution Split.
The model chooses a correct method but executes it wrongly (or vice versa) — a right-approach/wrong-answer or wrong-approach/right-answer split.
The car wash is a 5-minute walk away. Do you drive the car there, or walk?
Probes: Whether commonsense survives a prompt engineered to trigger the wrong pattern.
Trap: '5-minute walk' primes 'just walk' — models forget the whole point is to wash the car, which has to come.
On math-checkable cases, compare the stated method and the final numeric answer against an oracle; failure = correct method with wrong computed result, or an answer not entailed by the shown work.
Sets up the right integral, then arithmetic-errors the evaluation.
- Do arithmetic in code. Hand the model a calculator and ground the answer in what it returns.
- Assert units and dimensions separately from the value.
- Ask for method and answer as separate fields, then check they agree.
- Do not ask a model to verify its own output in the same turn — it confirms far more readily than it catches.
- tool-assisted computation
- verify final against method
One tab per model, newest first. Each card is that model against this mode under every condition on a single day. The mitigations above are the claim; the residual on each card is what actually survived the intervention.
loop 2026-09-10 · x-ai/grok-4.6 as served · closed weights · released 2026-01 · set s1.4-live · cases v1.7 · conditions v1.0 · ground truth as of 2026-08-09
| Condition | Failure rate | 95% interval | n | Trap | Answerable |
|---|---|---|---|---|---|
No context bare · The model alone, with no supplied context. | 0% | 0%–10% | n=50 · eff 35 | 0% (n=30) | 0% (n=20) |
Irrelevant context null · Length-matched irrelevant context. The control. | 0% | 0%–10% | n=50 · eff 35 | 0% (n=30) | 0% (n=20) |
General context recipe_a · Broad retrieval — the first passage the source returns, unreviewed. | 0% | 0%–10% | n=50 · eff 35 | 0% (n=30) | 0% (n=20) |
Curated context recipe_b · Hand-checked retrieval — the same passage plus the provenance chain behind it. | 0% | 0%–10% | n=50 · eff 35 | 0% (n=30) | 0% (n=20) |
intervals overlap — not separated
What a document being there did, before any of its content. Should be small.
intervals overlap — not separated
What the information itself did. This is the publishable claim.
Unchanged by context: 0% with none, 0% with the strongest supplied source. For a control mode that flat line is the result — it is what licenses reading movement elsewhere in this loop as grounding rather than as a document being present. Nothing survives at the best condition on the cases measured here.
- — tool-assisted computation
- — verify final against method