The index.
An open, versioned taxonomy of the ways language models fail. Each mode carries a definition, a signature you can recognise in the wild, and — where a deterministic detector exists — a measured failure rate per model. How a rate is made.
Making It Up
- Supply the source rather than asking the model to recall one — a real passage substantially reduces post-cutoff fabrication across the panel.measured
- Resolve every citation yourself. An HTTP or DOI check in your pipeline, never the model's word that a reference exists.
- Diff what the model says it did against your tool log. Narration is not evidence of an action.
- Make "no basis found" a passing answer in your own eval, or you are training over-assertion.
States a checkable fact with total confidence and no source
The model asserts a specific, checkable claim with high linguistic confidence but no retrievable basis — a fabrication that is not a citation per se, just a stated "fact."
Cites a source that does not exist
A citation the model offers does not resolve to a real, reachable source, or resolves to one that does not contain the cited content.
Says it searched or ran something when it did not
The model narrates having taken an action (searched, ran, read, called) that the execution log shows it did not take.
Misremembers what was said earlier in the conversation
The model misremembers, contradicts, or fabricates details of earlier turns that were explicitly established in the conversation.
Invents a match between two sets of columns
Given two or more supplied data structures, the model asserts a correspondence between their fields — a join key, a column mapping, a schema alignment — that is not established by the inputs, and proceeds as though the mapping were given rather than flagging it as an assumption.
Caving Under Pressure
- Never re-ask a question in the same context after disagreeing with the answer. Open a fresh context to re-check.
- Strip authority framing from user input before it reaches the model when the task is factual.
- Push back on correct answers as well as wrong ones. Without the bidirectional control you cannot tell a reasoner from a caver.
- Ask for the basis before you challenge the claim, not after — a cornered model manufactures one.
Abandons a correct answer as soon as you push back
The model abandons a correct answer when the user merely pushes back, without new evidence — measured with a bidirectional control so that flips on incorrect answers (a good thing) are not counted as failures.
Shifts its position to match what you seem to want
The model's substantive stance shifts to match a preference the user signals (praise-seeking), independent of a factual pushback.
Changes its answer when you claim to be an expert
The model's answer to a factual question changes based on the claimed authority/identity of the asker rather than the content.
Invents a number when asked where a figure came from
When asked to justify a specific claim, the model cannot supply a real basis and instead restates, deflects, or invents one.
Forgetting the Rules
- Restate hard constraints in the last message, not only in the system prompt.
- Validate structured output programmatically on every turn. Sampling turns is how schema drift ships.
- Carry a canary constraint through long sessions and alert when it decays.
- Compact before you extend. A shorter conversation holds a rule better than a longer window does.
Forgets a rule you set earlier in a long chat
A constraint set early in a long conversation is silently dropped as the dialogue continues.
Drops the required format when the task gets hard
A required output format degrades when the task content is cognitively demanding or adversarially distracting.
Breaks the JSON shape partway through a conversation
When asked for a structured output (JSON/schema), the model emits output that fails validation — especially after several turns.
Misses a fact buried in the middle of a long document
Retrieval of a specific fact from long context depends on where in the context the fact sits ("lost in the middle").
Word Plays & Traps
- Do arithmetic in code. Hand the model a calculator and ground the answer in what it returns.
- Assert units and dimensions separately from the value.
- Ask for method and answer as separate fields, then check they agree.
- Do not ask a model to verify its own output in the same turn — it confirms far more readily than it catches.
Picks the right method, then does the arithmetic wrong
The model chooses a correct method but executes it wrongly (or vice versa) — a right-approach/wrong-answer or wrong-approach/right-answer split.
Gives different answers to the same question on repeat
Repeated sampling of the same reasoning problem yields answers that disagree with each other.
Rubber-stamps its own wrong answer when asked to check
Asked to check its own answer, the model rubber-stamps a wrong result instead of finding the error.
Gets the number right and the unit wrong
The model produces a quantity with inconsistent or wrong units, or mixes unit systems, yielding a dimensionally invalid result.
Say It Differently
- Test every production prompt in three or more paraphrases before shipping. You ship a phrasing, not an intent.
- Version prompt text like code. A trailing space is a change.
- Sample repeatedly at your production temperature and measure the spread, not just the modal answer.
- Randomise option order anywhere the task is multiple-choice shaped.
Changes its answer over a typo or extra whitespace
Trivial surface perturbations (whitespace, typos, reordering of irrelevant clauses) change the substantive answer.
Same question, reworded, different answer
Meaning-preserving paraphrases of the same question yield materially different answers.
Picks a different option when you reorder the choices
For multiple-choice or ranking tasks, the chosen answer depends on the order options are presented.
Tool Discipline
- Validate every tool call against its schema before executing it.
- Cap iterations and detect repeated identical calls. Non-termination is a failure mode, not a hang.
- Diff the final answer against the tool return rather than trusting the summary of it.
- Replay the call log to compute true state. Never ask the model for the running total.
Reaches for the wrong tool for the job
Given multiple tools, the model routes a request to the wrong tool despite adequate labels/descriptions.
Calls a tool with arguments it will reject
The model calls a tool with arguments that violate its schema or references a tool/parameter that does not exist.
Loops on the same tool call without getting anywhere
The model repeats the same (or cyclic) tool calls without progressing toward the goal, failing to terminate.
Goes beyond, or contradicts, what the tool returned
The model's answer contradicts, ignores, or embellishes beyond what the tool actually returned.
Loses track of a running balance across steps
Across a sequence of state-changing tool calls, the model's running account of state diverges from what the calls actually produced.
Frozen in Time
- Inject the current date explicitly in context. Never assume the model knows what day it is.measured
- Ask for volatile facts to be stamped to the knowledge cutoff rather than accepted or refused outright.
- Route volatile fact classes to retrieval by default, and treat the routing as the mitigation.
- Record each model's published training cutoff in your own config. Staleness behaviour is uninterpretable without one.
States recent facts it has no way of knowing
For a fact that changed after the model's training cutoff, the model asserts a stale value as current instead of flagging uncertainty.
Sounds equally sure about stable and fast-moving facts
The model expresses uniform confidence regardless of whether a fact is stable or likely to have changed since training.
Gets date arithmetic wrong
Given an injected "current date," the model computes relative dates/intervals incorrectly.