Put an irrelevant document of the same length in front of the model and much of your retrieval win survives. That part was never retrieval.
the presence effect
The RAG placebo
Retrieval reduces fabrication. Some of that has nothing to do with what the document says.
Everyone measures retrieval the same way: ask without context, ask with your retrieved passage, report the difference as the value of retrieval.
That number is two effects added together, and only one of them is about information.
The debt this accumulates
Putting any document in a prompt changes behaviour. It changes length. It changes attention. It signals that a source exists and should be deferred to.
A model handed a length-matched passage about office stationery — parking permits, opening hours, a stationery order — already behaves differently from one handed nothing. On a question the passage says nothing about.
So if you are measuring with two arms, you cannot tell which world you are in — and the two worlds have opposite roadmaps.
Why it changes what you build
If most of your improvement is a presence effect, a better reranker buys you very little. The model is being made cautious by the existence of a document, not informed by its contents. The lever is the instruction that accompanies any retrieved passage.
If most of it is content, the opposite. Retrieval quality is everything and a passage that actually contains the answer is worth paying for.
When two arms is fine
If you only care whether the end-to-end system got better and you will never attribute the gain to a component, two arms answers your question. Ship it.
The moment someone asks "should we invest in the retriever or the prompt", two arms cannot answer and you will guess. I have watched teams spend a quarter on the wrong one.
Two arms is a comparison. Three is a measurement.
Have you ever run the irrelevant-document control? What happened?
Every figure here describes something measured and committed. See the measurements · read the method