Methodology.

The census measures, for every failure mode × model, a failure rate with a 95% Wilson confidence interval, computed from deterministic trials. Failure modes come from the AI Failure Mode Index, an open taxonomy published on this site. For where this sits relative to evaluator libraries like LangSmith, Inspect and HELM, see the framework.

The per-cell rates are the primary result. The CFMI cumulative index exists on top of them because a single comparable number is genuinely useful — but it is constructed so it cannot be quoted more precisely than the data supports, and it never replaces the cells.

CFMI — the cumulative failure index

CFMI is the severity-weighted mean failure rate across every mode a census measured, scaled to 0–100. Lower is better. 0 means the model failed no trial of any instrumented mode; 100 means it failed every trial of every mode.

The weights are not tuned constants — they come from each mode's own severity axes as published in the taxonomy, so a weight can be checked against the entry it came from. Harm dominates because it captures what the failure costs when it lands; prevalence follows; stealth is last, though an unnoticed failure is arguably the worst kind.

Four constraints make the number publishable rather than merely quotable:

1. Propagated uncertainty. Every cell carries binomial error; the index carries the weighted sum of it as a 95% interval. Where two models' intervals overlap, the site states that the index does not separate them instead of printing a false ordering.
2. Coverage gating. A CFMI over a subset of modes is not comparable to one over the full set. Coverage is reported on every row, and a model measured on a different mode set is excluded from the ranking rather than silently compared.
3. Unmeasured is not passed. A mode with no trials lowers coverage; it never counts as a success and so cannot flatter a score.
4. Always one click from the cells. Every index figure links to the per-mode rates and the transcripts underneath them.

What CFMI does not measure: capability, helpfulness, latency, cost, safety refusals, or anything outside the instrumented modes. It is a summary of a failure surface, not a quality score, and a model with a low CFMI can still be the wrong choice for a given task.

This census

No census has been published yet; panel, trial counts, and case-set versions appear here once one is.

Determinism & statistics

Temperature and token caps are fixed per run and recorded on the run. Each cell's denominator is PASS + FAIL only; NOT_APPLICABLE and ERROR outcomes are excluded from the rate and surfaced as anomalies. The interval is the Wilson 95% score interval.

Not yet instrumented (8 modes)

These are real failure modes the census does not yet score — each would otherwise measure case-authoring error rather than model error.

• fmi_1_1 Unsupported-Claim Confidence

  Open-domain fact-checking needs a general oracle we don't have; scoring would encode our labels.

• fmi_2_2 Flattery / Agreement Drift

  No crisp oracle for a 'wrongly' changed stance on subjective/recommendation tasks.

• fmi_3_2 Format-Lock Under Distraction

  Overlaps fmi_3_3 (schema validity), which is cleaner to instrument; folded in to avoid double-counting.

• fmi_4_2 Self-Consistency Spread

  Per-cell trial variance already reports sampling spread; a dedicated mode would double-instrument it.

• fmi_4_3 Verification-When-Asked Failure

  Cleanly seeding 'a wrong answer the model believes is its own' is hard to automate without confounds.

• fmi_5_1 Prompt-Perturbation Drift

  Overlaps fmi_5_2 (paraphrase invariance), a cleaner meaning-preserving perturbation.

• fmi_5_3 Option-Ordering Sensitivity

  Narrow to multiple-choice formats not central to the open-response panel.

• fmi_7_2 Recency-Confidence Miscalibration

  Requires eliciting and scoring calibrated confidence, which is format-dependent and easy to mis-instrument.

Limitations

Detectors and case sets are versioned and, at v1, contain generated ground truth that is human-reviewed before any full census. Oracles are conservative and can miss subtle failures; a failure rate is a lower-bound signal on a specific, instrumented behavior, not a general quality verdict.

Reproducing a run

Every run is defined by a config file (panel, modes, trials, temperature, case versions). Re-issuing the same census command resumes it — completed cells are skipped. See the disputes policy for reproducibility challenges.