Where this sits in the eval stack.
Evaluation tooling is crowded at the bottom of the stack and empty at the top. There are excellent libraries for running an eval and almost nothing that says which failures are worth running, what to call them, or how often frontier models actually commit them. That gap is what the Failure Mode Index fills.
The four layers
Read top-down: each layer consumes the one below it. Most teams own layers 3 and 4 and have nothing for layers 1 and 2 — so every team invents its own private failure vocabulary.
Answers: Which failure modes exist, what each one is called, how it is defined, and what its severity profile is. 27 catalogued modes across 7 classes, versioned and open.
ModelCensus FMI · MLCommons hazard taxonomies · NIST AI RMF (governance-level)
Answers: For each mode × model: a failure rate with a confidence interval, from a fixed panel, reproducible and with published transcripts. Plus a cumulative index that reports its own ties.
ModelCensus census runs + CFMI · HELM scenario coverage
Answers: Given an output and maybe a reference, produce a score: exact match, JSON validity, embedding distance, tool-trajectory correctness, pass@k, bootstrap intervals.
LangSmith / openevals · Inspect AI (UK AISI) · lm-evaluation-harness · Ragas · DeepEval · promptfoo
Answers: Run the calls, trace them, store datasets, gate CI, watch production. The plumbing every team needs and most already have.
LangSmith tracing · Braintrust · Langfuse · Arize / Phoenix · your own CI
The practical consequence: a team using LangSmith or Inspect has the machinery to detect a failure but no external reference for whether the rate they are seeing is normal. A 12% citation-fabrication rate is either alarming or unremarkable depending on the population baseline — and the baseline is what layers 1 and 2 publish.
How our detectors map to evaluator categories
The FMI harness is not a competing evaluator library — its detectors fall into the same categories the established libraries already use. If you know LangChain's string/comparison/trajectory split, you already know how these work.
| Evaluator category | FMI equivalent | Modes | How it decides |
|---|---|---|---|
| String / heuristic evaluators | Deterministic oracle detectors | fmi_4_1, fmi_4_4, fmi_6_5, fmi_7_3 | Exact-match, numeric tolerance, unit and arithmetic checks. No judge model. |
| Trajectory / agent evaluators | Tool-control detectors | fmi_6_1 – fmi_6_4 | Tool selection, argument validity, termination, and fidelity to tool output. |
| Pairwise / comparison evaluators | Consistency detectors | fmi_5_2, fmi_2_1 | Same question re-asked; disagreement across paraphrase or under pressure is the signal. |
| Groundedness / faithfulness | Attribution detectors | fmi_1_2, fmi_1_3, fmi_2_4 | Citations checked against the DOI registry; claimed tool actions checked against the log. |
| Hallucination detection | Knowledge-boundary detectors | fmi_7_1, fmi_7_3 | Calibration on volatile facts — measured as hedging, not as factual correctness. |
Four deliberate constraints
These are the choices that make the numbers defensible. Each one costs coverage, and each one is worth it.
Every detector instrumented today is a deterministic oracle: an exact answer, a schema, a registry lookup, a tool log. A judge model imports its own failure modes into the measurement, so it is the last resort rather than the default — and where one is ever used, the mode says so on its own page. The cost of that discipline is that only 19 of 27 modes are instrumented; the other 8 are listed with the reason and never scored.
When a check cannot be completed — a blocked request, a paywall, a timeout — the trial is recorded as not-applicable and leaves the denominator. Counting those as failures measures publisher bot policy, not the model. This single rule moved one of our published modes by 11 points.
Every cell and the index carry a 95% interval. Where two models' intervals overlap, the site states that the index does not separate them instead of printing a false ordering.
A run cannot publish until a human has read the transcript of every failed trial the site will display and approved it as genuine; the remainder are sampled at a disclosed rate. The publish step is hard-blocked, not warned about.
Using this with your own stack
Name failures with mode ids in your own reviews and incident reports. A shared taxonomy makes two teams' findings comparable.
Browse the 27 modes →Run your own eval, then check your rate against the published population rate for that mode and a comparable model.
See the matrix →The detectors ship as a standalone package with no app dependencies — adapters, detectors, case sets, runner and statistics.
Read the methodology →