If you cannot run it again next quarter and get a comparable number, you did not measure anything. You had an experience.
reproducibility
An eval that cannot be re-run is an anecdote
Four layers have to hold still for a number to mean the same thing next quarter. Each one has a cost.
Most retrieval evals I see are unrepeatable by construction. They query a live index at run time.
Every probe gets slightly different context, and next month's index is a different index. The number is real. It describes an event, not a system.
The four layers that have to hold still
Superpower: a mid-run upgrade cannot silently change how a loop is scored.
Why: Where it breaks: a fix you ship today waits for the next loop to take effect.
Superpower: every model sees exactly the same context. The passages are real retrieval output, captured once.
Why: Where it breaks: you are no longer measuring your retriever. That is a real loss, and I chose it deliberately.
Superpower: anyone can re-run the identical experiment from committed files a year from now.
Why: Where it breaks: volatile facts inside cases expire. Every patch has to be a version bump, never an edit in place.
Superpower: a change in score came from the model, not from the scorer.
Why: Where it breaks: a detector bug is pinned too. We once found one scoring refusals as assertions, and it had reversed a headline.
The verdict
For anything you will quote or compare over time: freeze all four. For evaluating a retriever: let retrieval run live — but then freeze the model, or you are back to two moving parts.
Nothing moves that you did not move on purpose. That sentence is the entire method.
Could you re-run last quarter's eval today and get a number you would be willing to compare?
Every figure here describes something measured and committed. See the measurements · read the method