Methodology.
Every rate is a failure mode × model measured by deterministic detectors, with a 95% Wilson interval at the effective sample size. Nothing is computed on top of them — no index, no severity-weighted mean. Where one figure is wanted, the census reports a count of modes an intervention reached.
No RPN, no detection score. S × O × D multiplies ordinal ratings — (2, 9, 5) and (9, 5, 2) both give 90. Detectability belongs to the system you deploy into, so it is out of scope, not estimated.
Understand the Failure Mode Index
27 named failure modes in 7 classes. You cannot measure a failure you have not named.
each with a definition, the question that exposes it, and its severity axes
a grouping of modes — never an axis, never a unit of measurement
~6 modes chosen to answer one question, crossing classes on purpose
A set carries treatment modes an intervention should reach and control modes it should not. Without controls, “it worked” and “it moved everything” look identical.
Build the context harness
Frozen prompts, frozen contexts, versioned together. Comparability comes from nothing moving that you did not move on purpose.
Only the second is a publishable claim about information.
Contexts are frozen text, never a query resolved at run time — a live query would make the delta partly a measurement of the retriever. One is a null control: irrelevant content, length-matched within 10%, same wrapper, same position.
- Case set
- versioned · frozen prompts · trap and answerable tagged
- Conditions
- versioned · one bare, one null, N recipes
- Detector
- one per mode, pinned per card
- Harness
- one version across a whole loop
Execute and benchmark
One call becomes a rate becomes a shape becomes a pivot. Four levels, each contained by the next.
One call. One model, one case, one context, scored by one detector.
- Records
- prompt sent + full response, verbatim
- Verdict
- PASS · FAIL · N/A · ERROR, from the detector
- Reason
- the detector's own line, never hand-written
- Publishes
- no — it is a log row
- — Every FAIL is reviewed by a person before publish.
Rate is FAIL / (PASS + FAIL). Not-applicable and errored probes leave the denominator rather than counting as passes, and surface as anomalies. Trap and answerable are reported apart, never pooled.
Repeats of one case are not independent, so every interval here is computed at the effective sample size n / (1 + (repeats − 1) × ICC), not at n. The intra-case correlation is measured per mode, 0.10 to 0.78. This page used to say the intervals were “slightly narrow” and that the fix was future work; they were narrow by a factor of three to four, and it was not slight.
No admin route, no login form, no password field. Review and publish happen on a command line; the site is generated from the committed manifests, so git log is the publication history. Draft loops show metadata only — never rates.
- — one served-on date across the loop
- — one harness version
- — one dimension varying per card
- — a null control behind every content effect
- — no percentage from a single observation
- — an as-of date on every case
- — a training cutoff on every class-7 mode
- — both tags reported when both are measured
- — every failure reviewed by a person
- — a residual written on every card
Reasoning cases & contamination
A trap looks like a familiar puzzle but resolves differently. Each names the correct answer and the wrong one it is built to elicit, so scoring is three-way: right, wrong, or took the planted lure.
Every trap ships with a matched control — same form, no lure. A trap rate beside its control rate is a claim about reasoning; alone it is a number.
The famous wordings are in every corpus, so asking them measures recall. Same structure, different quantities and framing.
A set printed here is in the next training corpus, so it is marked published. Held-out twins exist for the same structures.
Structure stays, surface changes, and the rotation date is recorded on the set.
Published rate falling while its held-out twin holds steady is the result — the models learned the test. Fresh cases would delete it.
reasoning cases run on their own case-set version and never enter a census set
Catalogued but not scored (9 modes)
Real failure modes the census does not score — each with its reason. Declining to measure what cannot be measured well is a choice, not an omission.
Not instrumented: this mode requires token-level log-probabilities, which the panel's providers do not uniformly expose — Anthropic models return none…
No crisp oracle for a 'wrongly' changed stance on subjective/recommendation tasks.
Overlaps fmi_3_3 (schema validity), which is cleaner to instrument; folded in to avoid double-counting.
Per-cell trial variance already reports sampling spread; a dedicated mode would double-instrument it.
Cleanly seeding 'a wrong answer the model believes is its own' is hard to automate without confounds.
Overlaps fmi_5_2 (paraphrase invariance), a cleaner meaning-preserving perturbation.
Narrow to multiple-choice formats not central to the open-response panel.
Not instrumented: this mode requires token-level log-probabilities, which the panel's providers do not uniformly expose — Anthropic models return none…
What a rate is not, and how to re-run one
Oracles are conservative and can miss subtle failures, so a rate is a lower-bound signal on one instrumented behaviour — never a quality verdict on a model. Ground truth at v1 is generated, and human-reviewed before any full census.
A run is its config: panel, modes, trials, temperature, case versions. Re-issuing the same census command resumes it and skips completed cells. Reproducibility challenges go through disputes.