Reliability engineering, for language models.
Every engineering discipline has a failure taxonomy. Generative AI doesn’t.
A systematic application of Failure Mode and Effects Analysis to language model behaviour — decoupling information utility from prompt-induced attention distortion, and measuring how far supplied context remediates each failure mode, model by model.
40 published loops · 20 models · 10 failure modes
Making It Up
Problem: Confident claims and citations with nothing real behind them.
Engineering guidance: Supply the source rather than asking the model to recall one — a real passage substantially reduces post-cutoff fabrication across the panel.
Context is not a general-purpose fix.
Supplying a source is the standard answer to almost every reliability complaint about language models. It is a real intervention with a narrow reach: it moves some failure modes a long way, leaves others untouched, and on a few it makes things worse.
Each axis here is one failure mode. The dotted outline is the model without context, fixed in place; the filled shape is the same panel with a source supplied. The gap between them is what context did — and the axes that do not move are as much of the result as the ones that do, because a fix that moved everything would not be grounding anything.
Knowing which of your failures context can reach, before you build a retrieval pipeline to reach them, is the difference between an intervention and a hope.
band = middle 80% of 20 models · 10 modes · set s1.4-live · spans 7 of 7 classes · axes are modes, never classes
Dotted = no-context baseline, fixed. The gap is the impact. Further out = fails more.
More context isn't more signal.
Standard evals treat context as additive, assuming more tokens equal better outputs. In reality context quality dictates information gain. Unfiltered volume introduces a quality penalty — distraction, context rot, and reasoning drag that eats away at accuracy.
The two axes are independent, and most evaluation collapses them into one. A longer window is not a better one; a denser passage is worth more than a larger corpus; and a retrieval step that returns something irrelevant has not failed to help, it has actively spent your budget to make the answer worse.
This census varies quality with volume held constant, which is what lets a change be attributed to what the context says rather than to how much of it there is. The vertical axis is a different experiment, and one worth running.
Model performance is a moving target.
A single failure mode spans 0% to 100% across top LLMs. Model IDs don’t guarantee static behaviour — quantization, routing, and hidden updates change performance daily.
Which means a rate you measured last quarter describes an endpoint that may no longer exist. Nor is recency a proxy: across three years of releases, where a model lands on a given failure mode is not recoverable from how new it is.
So every loop records the model actually served rather than the one requested, and a rate stays attached to the thing that produced it. Re-running months later is a new measurement, not a confirmation of the old one.
Every number has a path you can walk back.
Start at a rate and work back: the card it sits on, the individual probe that failed, and the raw text of what was sent and what came back. Each step is a click, and each one shows its working rather than summarising it.
The method is versioned alongside the results — cases, conditions and detectors move together, and publishing is a commit rather than a button, so git log is the publication history. Where something turns out to be wrong, the correction is a dated erratum beside the original rather than a quiet edit of it.