Real context moved -5.0pp on Cross-Turn Schema Validity.
Breaks the JSON shape partway through a conversation.
~intervals overlap — not separated from no change
When asked for a structured output (JSON/schema), the model emits output that fails validation — especially after several turns. The rule
Return a JSON object for a fictional person with fields name (string), age (integer), email (string).
```json { "name": "Sienna Voss", "age": 31, "email": "sienna.voss@rhythmstage.com" } ```
detector fmi_3_3_v1: all turns produced schema-valid JSON
```json { "name": "Sienna Hartwell", "age": 31, "email": "sienna.hartwell@example.com" } ```
detector fmi_3_3_v1: all turns produced schema-valid JSON
loop 2026-09-09 · anthropic/claude-sonnet-4.6 as served · closed weights · released 2026-02 · set s1.4-live · cases v1.7 · conditions v1.0 · ground truth as of 2026-08-09
Return a JSON object for a fictional person with fields name (string), age (integer), email (string).
expected as of 2026-08-09
Schema-valid JSON on all 3 turns. The third turn adds a detail with nowhere to put it — an extra key is the failure.
Nothing supplied. The baseline.
```json { "name": "Sienna Voss", "age": 31, "email": "sienna.voss@rhythmstage.com" } ```
fmi_3_3_v1
detector fmi_3_3_v1: all turns produced schema-valid JSON
0–80% · whisker = 95% interval
never pooled · a trap ratio would skew the delta
25% with no context, 0% under the best supplied source. What survives is 0 failing probes at that best condition. Nothing survives at the best condition on the cases measured here.
- schema-constrained decoding
- validate-and-repair loops
fmi_3_3_v1 · 5 repeats per case · best context curated context · rollups + residuals committed · probe log outside git · manifest hash ties them · method