The window grew by orders of magnitude. The method for deciding what goes in it did not change at all.
context inflation
Context windows grew a hundredfold. Evals did not.
We can fit a book in the prompt and still measure as though it were a paragraph. Here is the three-stage fix.
By the end of this you will be able to say how much of a context improvement came from what the context said — rather than how much of it there was.
Most RAG evals cannot. They compare no context against a retrieval dump, and at modern window sizes that comparison adds two effects together.
What you need
- Your question set
- A tokenizer, so you can count what each arm actually sends
- One irrelevant passage per question — boring, well-formed, the same length as the real one
Stage 1 — hold volume constant
Every arm that carries context gets the same token budget. In our census every such arm sits between 160 and 175 tokens.
Comparing no context against a 20,000-token retrieval dump.
Why: Any change is now volume and relevance mixed together, and you cannot attribute it to either.
Token counts across context arms within about ten percent of each other.
Stage 2 — add the arm nobody runs
A length-matched irrelevant passage. It tells you what a document does simply by being there.
Using an empty string, or a single sentence, as the control.
Why: Then the control differs in length as well as content, and you are back to two variables.
Stage 3 — score questions the context says nothing about
This is the one that surprised me. On our panel, paraphrase stability is near perfect when a question is asked plainly — and degrades sharply once an irrelevant document is in the prompt.
A retrieval step that returns something irrelevant has not failed to help. It has spent your budget to make the answer worse.
The accelerator
Freeze the passages once. Capture real retrieval output, commit it, and run every model against the identical text. You lose the ability to measure your retriever. You gain a delta you can actually attribute.
Does your RAG eval hold token count constant between arms — or is it quietly measuring volume?
Every figure here describes something measured and committed. See the measurements · read the method