Bearings have spalling, brinelling, fluting, cage fracture. We have one word and a vibe.
why taxonomies matter
One word, four bugs
"Hallucination" is doing the work of four distinct failures. Your RAG pipeline addresses two.
The standard advice is to pick a benchmark, track the number, and ship when it goes up.
That number is an average over failures that have nothing to do with each other.
What the average costs you
Ask a mechanical engineer how a bearing fails and you get a list: spalling, brinelling, fluting, cage fracture. Each has a name, a signature, a rate, and a known mitigation. Boring. Boring is what a mature field looks like.
Ask how a language model fails and you get 87.3%.
"Hallucination" is the worst offender. It covers inventing a citation, narrating a search that never ran, misremembering a number you supplied four turns ago, and asserting an unknowable figure with total confidence.
Four failures. Four different fixes. Your retrieval pipeline addresses two of them.
Where the popular advice is right
Benchmarks are genuinely good at one job: shortlisting. If you need three candidates from thirty, a capability score is the fastest filter that exists and I use them for exactly that.
They stop being useful the moment you have chosen, because the model that is strongest on average can be the worst on the specific mode that will hurt your product — and the ranking structurally cannot show you that.
The hard part
Naming failures is slow and unglamorous. We named twenty-eight and can only instrument nineteen; nine are catalogued with a written reason for why we will not score them.
That is the trade. A taxonomy is an asset that compounds across model generations. A leaderboard is a claim with a shelf life of about three months.
The failure you didn't enumerate is the one that reaches production.
What's the last incident your team had — and does it have a name you could track a regression against?
Every figure here describes something measured and committed. See the measurements · read the method