An eval that cannot be re-run is an anecdote
Frozen prompts, versioned cases, pinned detectors — and why a live retrieval query breaks all of it.
The most common way a retrieval eval becomes unreproducible is also the most natural way to build one: resolve the retrieval query at run time, against a live index.
Do that and every probe gets slightly different context. The measured difference between conditions is then partly a measurement of your retriever's variance on that day, and re-running next month measures a different index. The number is real and it describes an event, not a system.
Frozen does not mean fake
The passages are real retrieval output, captured once and then frozen — the first passage the source returns for the general condition, the same passage plus its provenance chain for the curated one. What is fixed is that every model sees exactly the same text.
The cost is that we are not measuring your retriever. The benefit is that the delta between conditions is attributable to the context and to nothing else, and that anyone can re-run the identical experiment from the committed files a year from now.
Every figure here describes something measured and committed. See the measurements · read the method