Failure Mode Index · fmi_4_2
Self-Consistency Spread.
Reasoning & Calculationnot instrumented · status: proposed
Repeated sampling of the same reasoning problem yields answers that disagree with each other.
The question that exposes it
Solve this word problem five times. Do you get the same answer each time?
Probes: Whether repeated sampling of one problem agrees with itself.
Trap: At temperature, answers scatter — the model isn't sure what it thinks.
Signature
Answer entropy across N samples at fixed temperature.
Example
Five runs of one word problem give three different totals.
Severity axes
prevalence
medium
harm
medium
stealth
medium
Engineering guidance · class 4
- Do arithmetic in code. Hand the model a calculator and ground the answer in what it returns.
- Assert units and dimensions separately from the value.
- Ask for method and answer as separate fields, then check they agree.
- Do not ask a model to verify its own output in the same turn — it confirms far more readily than it catches.
Mitigations
- self-consistency voting
- lower-variance decoding
Measurement
This mode is catalogued but not instrumented: it has no reliable deterministic detector yet, so no loop scores it. See /methodology for the exclusion reasons.