Failure Mode Index · fmi_3_4
Needle-Position Sensitivity.
Instruction Adherence & Long-Contextinstrumented · tier 2 · status: stable
Retrieval of a specific fact from long context depends on where in the context the fact sits ("lost in the middle").
The question that exposes it
Somewhere in this 60-page document is a vault code. What is it?
Probes: Whether recall depends on where the needle sits (lost-in-the-middle).
Trap: Facts at the start and end are recalled; the middle vanishes.
Signature
The same needle swept across positions (start/25%/50%/75%/end) of a long filler context; failure = recall that drops at particular positions.
Example
Recalls a planted code at the top and bottom but not the middle of a 60k-token context.
Severity axes
prevalence
high
harm
medium
stealth
medium
Engineering guidance · class 3
- Restate hard constraints in the last message, not only in the system prompt.
- Validate structured output programmatically on every turn. Sampling turns is how schema drift ships.
- Carry a canary constraint through long sessions and alert when it decays.
- Compact before you extend. A shorter conversation holds a rule better than a longer window does.
Mitigations
- position-robust attention
- retrieval over raw stuffing
Measurement
No published loop has measured this mode yet. When one does, its cards appear here, newest first.