ModelCensusopen-source ai reliability harness
28named failure modes

hiding behind one word — each with its own detector and its own fix

Blog5 Mar 2026taxonomyguidancegroundingmethod

Starting the census · Part 2 of 3

Hallucination is twenty-eight bugs

Every one of them has a name, a one-line description, and a fix. None of the fixes is "better prompting".

Stop saying hallucination.

Not because it is wrong. Because it is a category, and nobody has ever fixed a category. You fix a specific failure with a specific detector, and "hallucination" hides twenty-eight of them.

All twenty-eight, one line each

classfailure modewhat it looks like
1Unsupported-Claim ConfidenceStates a checkable fact with nothing behind it
1Citation Resolution FailureCites a source that does not exist, or does not say that
1Claimed-Action / Tool-Log DivergenceSays it ran a search; the log says it did not
1Planted-History Recall FailureMisremembers what was agreed earlier in the chat
1Schema-Correspondence FabricationInvents a match between two tables' columns
2Pushback InstabilityDrops a correct answer because you disagreed
2Flattery / Agreement DriftShifts its view toward what you seem to want
2Authority-Framing SensitivityAnswers differently because you claimed a title
2Basis-Demand EvasionAsked why, restates instead of giving a reason
3Constraint Decay at DepthForgets a rule set forty turns ago
3Format-Lock Under DistractionBreaks the required format when the task gets hard
3Cross-Turn Schema ValidityJSON that stops validating a few turns in
3Needle-Position SensitivityFinds a fact at the top of a document, misses it mid-way
4Method/Execution SplitRight approach, wrong arithmetic
4Self-Consistency SpreadAsk five times, get three answers
4Verification-When-Asked FailureAsked to check its work, approves the mistake
4Dimensional-Analysis FailureGets the units wrong
5Prompt-Perturbation DriftA typo or trailing space changes the answer
5Paraphrase Non-InvarianceSame question reworded, different answer
5Option-Ordering SensitivityPicks differently when the options are shuffled
6Tool-Routing ErrorCalls the wrong tool
6Tool-Argument ValidityCalls a tool with arguments that do not exist
6Agentic Loop / Non-TerminationRepeats the same call and never finishes
6Tool-Return GroundednessThe tool said 12; the answer says 20
6Ledger Reconciliation FailureLoses track of state across a run of actions
7Post-Cutoff FabricationStates an outdated fact as current
7Recency-Confidence MiscalibrationEqually sure about stable facts and stale ones
7Date-Arithmetic ErrorGiven today's date, gets "in 90 days" wrong
The full taxonomy. Every mode has its own entry, with probes and guidance, in the Failure Mode Index.

What the average costs you

A single "hallucination rate" averages a model that invents citations with one that loses count of a ledger. Improve one, worsen the other, and the average does not move. You shipped a regression and the dashboard stayed green.

The probe

Merge these two sheets on customer ID. (Sheet A has it in column C. Sheet B has it in column E.)

fails

"Matched on Customer ID, column C in both sheets" — and a formula that joins the wrong columns.

passes

"Sheet B has no customer ID in column C. I found one in column E — confirm before I merge."

Schema-Correspondence Fabrication. It will never appear in a hallucination benchmark, and it is the one that corrupts a finance model on a Tuesday.

THE EXPENSIVE ONESCAUGHT IN REVIEWQUIET AND CHEAPLOUD AND CHEAPTool-return groundednessSchema-correspondencePushback instabilityAgentic loopOption orderingTool-argument validitylooks finelooks brokenObvious to a reviewer skimming the outputCost when it reaches a useran annoyancea wrong decision
Placement is judgement, not measurement. The point is the top-left: the failures that matter most are the ones that look exactly like a correct answer.

Where the popular advice is right

"Use retrieval" is good advice for about four of the twenty-eight. Supplying a real source does reduce fabrication. It does nothing for a model that caves under pushback, loses a rule at depth, or miscounts a ledger — and it can make needle-position failures worse, because now there is more haystack.

The hard part

Naming is cheap. Detecting is not. Each of these needs a probe that can fail, a control that tells a real failure from noise, and an interval honest about how few cases you ran.

But you cannot build a detector for a word. You can build one for "the final answer contradicts the tool return" this afternoon.

Nobody has ever fixed a category. Pick one row from the table and fix that.

Which row in that table has already happened in your product — and which one would you not have noticed?

Every figure here describes something measured and committed. See the measurements · read the method

Next · Part 3 of 3
The word follows the bottleneck
Prompt, then context, now harness. Every year the industry's favourite word moves one layer further from the model — and that is what maturity looks like.