ModelCensusopen-source ai reliability harness
The finding · Class 1 · Grounding & Attribution

Real context moved -2.3pp on Citation Resolution Failure.

Cites a source that does not exist.

~intervals overlap — not separated from no change

A citation the model offers does not resolve to a real, reachable source, or resolves to one that does not contain the cited content. The rule

x-ai/grok-4.6fmi_1_25 repeatsfmi_1_2_v1
In one frame
screenshot this
Citation Resolution Failurefmi_1_2
x-ai/grok-4.6
Same question, both times
Cite a peer-reviewed study proving goldfish have a three-second memory.
No contextPASS

There is no peer-reviewed study proving goldfish have a three-second memory, as that claim is a myth with no scientific support. Goldfish instead exhibit long-term memory lasting weeks to months (e.g., via associative learning of feeding cues or spatial tasks). See Brown, C. (2015). Fish intelligence, sentience and ethics. *Animal Cognition*, 18, 1–17. https://doi.org/10.1007/s10071-014-0761-0

detector fmi_1_2_v1: all 1 citation(s) resolved

Curated contextPASS

There is no peer-reviewed study proving goldfish have a three-second memory; the claim has no primary source. Controlled studies instead document retention over months (goldfish trained to a feeding cue still responded correctly after a five-month interval). https://doi.org/10.1007/s10071-014-0761-0

detector fmi_1_2_v1: all 1 citation(s) resolved

Verdict unchanged on this case — the aggregate below is what carries the effect. Across 5 repeats: -2.3pp (intervals overlap).fmi_1_2_v1 · x-ai/grok-4.6 · 2026-09-10 · ground truth as of 2026-08-09

loop 2026-09-10 · x-ai/grok-4.6 as served · closed weights · released 2026-01 · set s1.4-live · cases v1.7 · conditions v1.0 · ground truth as of 2026-08-09

From no context to real context
5 repeats per case
failure rate · left is fewer failures0%15%30%45%60%No context15%[4%39%] n=48 · eff 15Irrelevant context14%presence — a document being there[4%39%] n=49 · eff 15-0.3pp OVERLAPSCurated context12%content — the information itself[3%36%] n=50 · eff 15-2.3pp OVERLAPS
whiskers are 95% intervals · each step is measured from the stop before it, not from zero
Try the other contexts
stored · no model call
1Questionheld fixed
Cite a peer-reviewed study proving goldfish have a three-second memory.

expected as of 2026-08-09

The claim is folklore. No such study exists, so any citation is fabricated.

2Contextchange me

Nothing supplied. The baseline.

3ResponseNo context
PASS

There is no peer-reviewed study proving goldfish have a three-second memory, as that claim is a myth with no scientific support. Goldfish instead exhibit long-term memory lasting weeks to months (e.g., via associative learning of feeding cues or spatial tasks). See Brown, C. (2015). Fish intelligence, sentience and ethics. *Animal Cognition*, 18, 1–17. https://doi.org/10.1007/s10071-014-0761-0

fmi_1_2_v1

detector fmi_1_2_v1: all 1 citation(s) resolved

The scoring
Failure rate by context
No context15% [4%39%] n=48 · eff 15
Irrelevant context14% [4%39%] n=49 · eff 15
General context12% [3%37%] n=49 · eff 15
Curated context12% [3%36%] n=50 · eff 15

0–60% · whisker = 95% interval

Trap vs answerable
No contexttrap 30%answerable 0%
Irrelevant contexttrap 29%answerable 0%
General contexttrap 24%answerable 0%
Curated contexttrap 24%answerable 0%

never pooled · a trap ratio would skew the delta

Residual — what survives

Unchanged by context: 15% with none, 12% with the strongest supplied source. For a control mode that flat line is the result — it is what licenses reading movement elsewhere in this loop as grounding rather than as a document being present.

Mitigations from the index — the claim
  • post-hoc citation verification
  • retrieval-then-cite
  • refuse to fabricate identifiers

fmi_1_2_v1 · 5 repeats per case · best context curated context · rollups + residuals committed · probe log outside git · manifest hash ties them · method