Context Sensitivity

What context does to each failure mode.

Every rate is measured under four arms — no context, an irrelevant document, and two supplied sources. The irrelevant one is the control: it separates information helping from a document merely being there.

How far a supplied source reaches, mode by mode, for every model measured on the same set under the same conditions.

Each row is one failure mode — a specific way an answer goes wrong, named so it can be argued with. Each column is a model. A cell is the change in how often that model fails that way once it is handed a real source, against being handed an irrelevant one. Green means the source helped. Red means it made things worse. Hatched means the intervals overlap, so the direction is real but the size is not established.

set s1.4-live · conditions v1.0 · 20 loops · delta is irrelevant context → best real context, in percentage points

anthropic/claude-3-haiku

anthropic/claude-haiku-4.5

anthropic/claude-sonnet-4.6

deepseek/deepseek-chat-v3.1

google/gemini-2.5-flash

google/gemini-2.5-flash-lite

google/gemma-3-27b-it

meta-llama/llama-3.1-8b-instruct

meta-llama/llama-3.3-70b-instruct

meta-llama/llama-4-maverick

microsoft/phi-4

mistralai/mistral-small-3.2-24b-instruct

openai/gpt-3.5-turbo

openai/gpt-4o

openai/gpt-4o-mini

openai/gpt-5-mini

openai/o4-mini

qwen/qwen-2.5-72b-instruct

qwen/qwen3-235b-a22b-2507

x-ai/grok-4.6

Citation Resolution FailureCites a source that does not exist

-10

-7

-12

-23

-38

-8

-15

-10

-1

-10

-6

-8

-20

-0

-16

-4

-2

Basis-Demand EvasionInvents a number when asked where a figure came from

+18

-5

-28

+20

+22

+40

+45

-8

-18

-3

-5

-23

-15

-8

Constraint Decay at DepthForgets a rule you set earlier in a long chat

+13

-100

-44

+80

-7

Cross-Turn Schema ValidityBreaks the JSON shape partway through a conversation

-5

+10

-5

+25

+15

Method/Execution SplitPicks the right method, then does the arithmetic wrong

-2

-4

-10

Dimensional-Analysis FailureGets the number right and the unit wrong

-2

Paraphrase Non-InvarianceSame question, reworded, different answer

+30

-43

-37

Ledger Reconciliation FailureLoses track of a running balance across steps

-13

-8

Post-Cutoff FabricationStates recent facts it has no way of knowing

-10

-6

-10

-22

-36

-70

-48

-12

-14

-18

-16

+18

-2

-6

-14

-18

-16

Date-Arithmetic ErrorGets date arithmetic wrong

-10

+10

-15

-5

-3

-13

-10

-3

-10

-3

shade = |pp| · teal = fewer failures · red = worse · hatched = intervals overlap

Failure mode	anthropic/claude-3-haiku	anthropic/claude-haiku-4.5	anthropic/claude-sonnet-4.6	deepseek/deepseek-chat-v3.1	google/gemini-2.5-flash	google/gemini-2.5-flash-lite	google/gemma-3-27b-it	meta-llama/llama-3.1-8b-instruct	meta-llama/llama-3.3-70b-instruct	meta-llama/llama-4-maverick	microsoft/phi-4	mistralai/mistral-small-3.2-24b-instruct	openai/gpt-3.5-turbo	openai/gpt-4o	openai/gpt-4o-mini	openai/gpt-5-mini	openai/o4-mini	qwen/qwen-2.5-72b-instruct	qwen/qwen3-235b-a22b-2507	x-ai/grok-4.6
Citation Resolution Failure fmi_1_2 · class 1 · Grounding & Attribution	+7.6pp overlaps	-10.3pp overlaps	-7.2pp overlaps	0.0pp overlaps	+1.9pp overlaps	-12.3pp overlaps	-23.1pp overlaps	-37.5pp overlaps	-7.7pp overlaps	-15.4pp overlaps	-10.0pp overlaps	-0.6pp overlaps	-10.2pp overlaps	-6.1pp overlaps	-8.1pp overlaps	-20.0pp overlaps	-0.3pp overlaps	-16.0pp overlaps	-3.7pp overlaps	-2.3pp overlaps
Basis-Demand Evasion fmi_2_4 · class 2 · Sycophancy & Epistemic Integrity	0.0pp overlaps	+17.5pp overlaps	-5.0pp overlaps	-27.5pp overlaps	+20.0pp overlaps	+20.0pp overlaps	+21.9pp overlaps	+40.0pp overlaps	+39.5pp overlaps	+45.0pp overlaps	-7.5pp overlaps	-18.2pp overlaps	-2.5pp overlaps	-2.5pp overlaps	-5.0pp overlaps	-23.1pp overlaps	-15.3pp overlaps	0.0pp overlaps	+7.5pp overlaps	-7.5pp overlaps
Constraint Decay at Depth fmi_3_1 · class 3 · Instruction Adherence & Long-Context	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	+13.3pp overlaps	-100.0pp separated	0.0pp overlaps	-43.8pp overlaps	0.0pp overlaps	+80.0pp overlaps	-6.7pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps
Cross-Turn Schema Validity fmi_3_3 · class 3 · Instruction Adherence & Long-Context	+5.0pp overlaps	0.0pp overlaps	-5.0pp overlaps	0.0pp overlaps	+5.0pp overlaps	0.0pp overlaps	0.0pp overlaps	+10.0pp overlaps	0.0pp overlaps	0.0pp overlaps	+5.0pp overlaps	-5.0pp overlaps	+25.0pp overlaps	+15.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps
Method/Execution Split fmi_4_1 · class 4 · Reasoning & Calculation	0.0pp overlaps	-2.0pp overlaps	0.0pp overlaps	-4.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	+6.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	-10.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps
Dimensional-Analysis Failure fmi_4_4 · class 4 · Reasoning & Calculation	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	-2.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	-2.0pp overlaps	0.0pp overlaps
Paraphrase Non-Invariance fmi_5_2 · class 5 · Robustness & Consistency	+30.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	-43.3pp overlaps	-36.7pp overlaps	+3.3pp overlaps	+3.3pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps
Ledger Reconciliation Failure fmi_6_5 · class 6 · Tool Use & Agentic Control	-12.5pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	-7.5pp overlaps	0.0pp overlaps	0.0pp overlaps	+2.5pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps
Post-Cutoff Fabrication fmi_7_1 · class 7 · Temporal & Knowledge Boundary	-10.0pp overlaps	-6.0pp overlaps	-10.0pp overlaps	-22.0pp overlaps	-36.0pp overlaps	-70.0pp separated	0.0pp overlaps	-48.0pp overlaps	-12.0pp overlaps	-14.0pp overlaps	+6.0pp overlaps	0.0pp overlaps	-18.0pp overlaps	-16.0pp overlaps	+18.0pp overlaps	-2.0pp overlaps	-6.0pp overlaps	-14.0pp overlaps	-18.0pp overlaps	-16.0pp overlaps
Date-Arithmetic Error fmi_7_3 · class 7 · Temporal & Knowledge Boundary	0.0pp overlaps	+5.0pp overlaps	0.0pp overlaps	-10.0pp overlaps	0.0pp overlaps	+10.0pp overlaps	-15.0pp overlaps	-5.0pp overlaps	+5.0pp overlaps	0.0pp overlaps	-2.5pp overlaps	-12.5pp overlaps	-10.0pp overlaps	-2.5pp overlaps	0.0pp overlaps	0.0pp overlaps	0.0pp overlaps	-10.0pp overlaps	-2.5pp overlaps	0.0pp overlaps

Separates on at least one mode: google/gemini-2.5-flash-lite (1 of 10) · meta-llama/llama-3.3-70b-instruct (1 of 10).

Cell: irrelevant context → best real context, in pp
Why not from no context: that would fold in the presence effect
Models: alphabetical · delta sortable, absolute rate not
No total, no mean, no index: modes are not commensurable
Absolute rates: on hover, and on the loop page
Empty cell: no published loop on set s1.4-live · stays empty

Method

Every loop is a recording · nothing here calls a model · how a loop runs