The index.
An open, versioned taxonomy of the ways language models fail — each mode with a definition, a signature you can recognise in the wild, and, where a deterministic detector exists, a measured failure rate per model. Modes without a reliable detector are listed and never scored.
Making It Up
The model asserts a specific, checkable claim with high linguistic confidence but no retrievable basis — a fabrication that is not a citation per se, just a stated "fact."
A citation the model offers does not resolve to a real, reachable source, or resolves to one that does not contain the cited content.
The model narrates having taken an action (searched, ran, read, called) that the execution log shows it did not take.
The model misremembers, contradicts, or fabricates details of earlier turns that were explicitly established in the conversation.
Caving Under Pressure
The model abandons a correct answer when the user merely pushes back, without new evidence — measured with a bidirectional control so that flips on incorrect answers (a good thing) are not counted as failures.
The model's substantive stance shifts to match a preference the user signals (praise-seeking), independent of a factual pushback.
The model's answer to a factual question changes based on the claimed authority/identity of the asker rather than the content.
When asked to justify a specific claim, the model cannot supply a real basis and instead restates, deflects, or invents one.
Forgetting the Rules
A constraint set early in a long conversation is silently dropped as the dialogue continues.
A required output format degrades when the task content is cognitively demanding or adversarially distracting.
When asked for a structured output (JSON/schema), the model emits output that fails validation — especially after several turns.
Retrieval of a specific fact from long context depends on where in the context the fact sits ("lost in the middle").
Word Plays & Traps
The model chooses a correct method but executes it wrongly (or vice versa) — a right-approach/wrong-answer or wrong-approach/right-answer split.
Repeated sampling of the same reasoning problem yields answers that disagree with each other.
Asked to check its own answer, the model rubber-stamps a wrong result instead of finding the error.
The model produces a quantity with inconsistent or wrong units, or mixes unit systems, yielding a dimensionally invalid result.
Say It Differently
Trivial surface perturbations (whitespace, typos, reordering of irrelevant clauses) change the substantive answer.
Meaning-preserving paraphrases of the same question yield materially different answers.
For multiple-choice or ranking tasks, the chosen answer depends on the order options are presented.
Tool Discipline
Given multiple tools, the model routes a request to the wrong tool despite adequate labels/descriptions.
The model calls a tool with arguments that violate its schema or references a tool/parameter that does not exist.
The model repeats the same (or cyclic) tool calls without progressing toward the goal, failing to terminate.
The model's answer contradicts, ignores, or embellishes beyond what the tool actually returned.
Across a sequence of state-changing tool calls, the model's running account of state diverges from what the calls actually produced.
Frozen in Time
For a fact that changed after the model's training cutoff, the model asserts a stale value as current instead of flagging uncertainty.
The model expresses uniform confidence regardless of whether a fact is stable or likely to have changed since training.
Given an injected "current date," the model computes relative dates/intervals incorrectly.