ModelCensusopen-source ai reliability harness
Failure Mode Index · fmi_1_5

Schema-Correspondence Fabrication.

Grounding & Attributionnot instrumented · status: proposed

Given two or more supplied data structures, the model asserts a correspondence between their fields — a join key, a column mapping, a schema alignment — that is not established by the inputs, and proceeds as though the mapping were given rather than flagging it as an assumption.

Why this is not scored

Proposed at v0.9.1 and not yet scored. The oracle is exact for synthetic file pairs with a known-correct mapping, so this is instrumentable by rule rather than by judgement — but the interesting half is the ambiguous pairs, where the correct behaviour is to ask rather than to map, and scoring "should have asked" needs a case design that does not simply reward refusal.

Signature

A stated field mapping that the supplied headers, types or sample rows do not license; downstream formulas, joins or transforms inherit it silently.

Example

Asked to merge two exports, the model maps `cust_id` to `CustomerRef` without evidence the two are the same key, and writes a lookup against it with no hedge.

Severity axes
prevalence
high
harm
high
stealth
high
Engineering guidance · class 1
  • Supply the source rather than asking the model to recall one — a real passage substantially reduces post-cutoff fabrication across the panel.measured
  • Resolve every citation yourself. An HTTP or DOI check in your pipeline, never the model's word that a reference exists.
  • Diff what the model says it did against your tool log. Narration is not evidence of an action.
  • Make "no basis found" a passing answer in your own eval, or you are training over-assertion.
Mitigations
  • state the join key explicitly rather than letting it be inferred
  • require the mapping as a separate reviewable output before any transform
  • supply headers, types and sample rows, not headers alone
  • treat an unhedged mapping as a failure in your own eval
Measurement

This mode is catalogued but not instrumented: it has no reliable deterministic detector yet, so no loop scores it. See /methodology for the exclusion reasons.