Ledger Reconciliation Failure.
Across a sequence of state-changing tool calls, the model's running account of state diverges from what the calls actually produced.
Opening balance 100. +50, −20, +30, −15. What's the final balance?
Probes: Whether running state matches a replay of the transactions.
Trap: The model's stated balance drifts from what the ledger actually implies.
Replay the tool call/return log to compute true final state; failure = the model's stated final state ≠ replayed state.
After three transfers the model reports a balance the ledger does not support.
- Validate every tool call against its schema before executing it.
- Cap iterations and detect repeated identical calls. Non-termination is a failure mode, not a hang.
- Diff the final answer against the tool return rather than trusting the summary of it.
- Replay the call log to compute true state. Never ask the model for the running total.
- external state tracking
- reconcile before reporting
One tab per model, newest first. Each card is that model against this mode under every condition on a single day. The mitigations above are the claim; the residual on each card is what actually survived the intervention.
loop 2026-09-10 · x-ai/grok-4.6 as served · closed weights · released 2026-01 · set s1.4-live · cases v1.7 · conditions v1.0 · ground truth as of 2026-08-09
| Condition | Failure rate | 95% interval | n | Trap | Answerable |
|---|---|---|---|---|---|
No context bare · The model alone, with no supplied context. | 0% | 0%–20% | n=40 · eff 15 | 0% (n=25) | 0% (n=15) |
Irrelevant context null · Length-matched irrelevant context. The control. | 0% | 0%–20% | n=40 · eff 15 | 0% (n=25) | 0% (n=15) |
General context recipe_a · Broad retrieval — the first passage the source returns, unreviewed. | 0% | 0%–20% | n=40 · eff 15 | 0% (n=25) | 0% (n=15) |
Curated context recipe_b · Hand-checked retrieval — the same passage plus the provenance chain behind it. | 0% | 0%–20% | n=40 · eff 15 | 0% (n=25) | 0% (n=15) |
intervals overlap — not separated
What a document being there did, before any of its content. Should be small.
intervals overlap — not separated
What the information itself did. This is the publishable claim.
Unchanged by context: 0% with none, 0% with the strongest supplied source. For a control mode that flat line is the result — it is what licenses reading movement elsewhere in this loop as grounding rather than as a document being present. Nothing survives at the best condition on the cases measured here.
- — external state tracking
- — reconcile before reporting