Add one arm to your RAG eval. It takes an afternoon.
A length-matched irrelevant document. Without it you cannot tell grounding from stuffing.
By the end of this you will be able to say which half of your retrieval win is information and which half is a document simply being in the prompt. Most teams cannot currently separate these.
What you need
- Your existing eval set, unchanged
- One irrelevant passage, length-matched to your real retrieved context within about 10%
- The ability to run the same questions three times instead of twice
Stage 1 — write the null control
Take a passage with nothing to do with your domain. Office admin notices work well: opening hours, parking permits, a stationery order. Match the token count of your typical retrieved chunk.
Do not use lorem ipsum or random tokens. You are testing what a plausible-but-irrelevant document does, not what garbage does.
Why: Real retrieval failures return coherent text about the wrong thing. That is the condition you are trying to reproduce.
Token count within 10% of your real context, same wrapper, same position in the prompt.
Why: If the control is shorter you have confounded length with relevance and the whole exercise is void.
Stage 2 — run three arms
No context. Null control. Your real retrieval. Same questions, same temperature, same everything else.
Do not resolve the retrieval query live at run time.
Why: Every probe gets slightly different context and your delta becomes partly a measurement of your retriever's variance that day. Freeze the passages.
Stage 3 — read the two steps separately
Two steps, two different claims. Only the second is about information.
If your control moved as far as your real passage, you do not have a grounding result. You have a prompt-length result.
Why: This is the outcome nobody wants and the one worth knowing. It changes what you build next — instruction tuning rather than a better reranker.
The accelerator
You do not need a new harness. Add one condition to whatever loop you already run, and reuse the same irrelevant passage across every question — it is a control, not a test case, and reusing it removes one more source of variance.
One passage, one extra arm, one afternoon. What is your presence-to-content split?
Every figure here describes something measured and committed. See the measurements · read the method