Your benchmark is stale before it publishes
Our panel spans three years of releases. Half of it shipped in one year.
The signal most evaluation has not absorbed: the gap between a model and its successor is now measured in months, and nearly every eval is built on the assumption that the thing you measured will still be there when you publish.
The forces
Competitive cadence is the obvious one. The less obvious one is that a model id is not a fixed artefact — behind it sit providers whose quantization and configuration differ, and both the weights and the routing change without the id changing.
So the thing you measured can move without anyone announcing a release at all.
Old way, new way
Obsolete: benchmark-a-generation as the unit of work. By the time it publishes it describes something that has been superseded twice.
Valuable: a fixed set of cases, conditions and detectors versioned together, so a new model is one more loop rather than a new experiment.
The timeline
Immediate: teams re-run benchmarks more often and notice the numbers move without explanation.
Near-term: those movements get attributed to model quality rather than to routing, because almost nobody records which provider actually served a request.
Dominant: provenance becomes the differentiator. Not the score — the ability to say what produced it.
What to do now
One field. It costs nothing and it is the difference between a contradiction and a new observation when you re-run in six months.
Why: Without it, a changed number looks like your eval is unstable rather than like the endpoint moved.
A leaderboard is a claim with a shelf life. A failure taxonomy is a claim about the shape of the problem.
How would you know if the model behind your API id changed tomorrow?
Every figure here describes something measured and committed. See the measurements · read the method