Seven classes. Twenty-eight named failure modes. "It hallucinated" is not one of them.
the ModelCensus taxonomy
Starting the census · Part 1 of 3
Seven ways a model fails
Every AI incident report uses one word. It is doing the work of seven different failures, and they need seven different fixes.
Most AI incident reports I read use one word: hallucinated.
That word covers a model inventing a citation, a model dropping a correct answer because someone disagreed, and a model reporting 20 degrees when its own tool said 12. Those are different failures with different fixes, and a word that covers all of them fixes none.
Engineering solved this problem decades ago. It is called Failure Mode and Effects Analysis: you name every way a component can fail before you argue about how often it does. So I applied it to language models.
Making It Up
Problem: Confident claims and citations with nothing real behind them.
Engineering guidance: Supply the source rather than asking the model to recall one — a real passage substantially reduces post-cutoff fabrication across the panel.
The seven classes, plainly
Confident claims and citations with nothing real behind them. Includes the model saying it searched when the log shows it never did.
Why: Resolve every citation yourself, and diff what the model says it did against your tool log. Narration is not evidence of an action.
Abandoning a correct answer because the user pushed back, flattered, or claimed to be an expert.
Why: Never re-ask in the same context after disagreeing. And test pushback on correct answers too, or you cannot tell a reasoner from a pushover.
A constraint set early quietly evaporates as the conversation gets longer or the task gets harder.
Why: Restate hard constraints in the latest message, not only the system prompt, and validate structured output on every turn.
Right method, wrong execution. Or the reverse. The answer has the right shape and the wrong number.
Why: Do arithmetic in code, and ask for method and answer as separate fields so you can check they agree.
The same question, reworded or reordered, gets a different answer.
Why: Test every production prompt in three paraphrases before shipping it. You ship a phrasing, not an intent.
The wrong tool, an invalid call, a loop that never ends, or a final answer that ignores what the tool returned.
Why: Validate every call against its schema before running it, cap iterations, and compute state from the call log rather than asking the model for it.
Outdated facts stated as current, and date arithmetic against a calendar the model cannot see.
Why: Inject today's date explicitly, and ask for volatile facts to be stamped to the training cutoff rather than asserted or refused.
My default setup
Before a model goes near anything that matters, I ask one question per class. Not "is it good" — that question has no answer. These do:
- Does every citation resolve, and does it say what the sentence claims?
- If a user disagrees with a correct answer, does it hold?
- Is the rule from turn one still being followed at turn forty?
- Is the arithmetic done in code, or in the model's head?
- Does the answer survive a paraphrase?
- Does the final answer match what the tool actually returned?
- Does it know what day it is, and does it say when a fact might be stale?
You cannot fix a failure you can only describe with a feeling. Name it first; the rate comes second.
Think of the last AI incident your team wrote up. Which of the seven was it — and did the write-up say so?
Every figure here describes something measured and committed. See the measurements · read the method