Blog5 Aug 2026conceptmethod

The model you tested is not the model you deployed

A model id is not a model. It is a routing decision that changes underneath you.

Benchmarks report results against a model name, as though the name identified a fixed artefact. It does not. Behind one id sits a routing layer that may send your request to different providers with different quantization and different configuration, and both the weights behind the id and the routing to it change over time.

Three of these four can change without the id changing. A result is a measurement of an endpoint on a date, not a permanent property of a model.

What we record instead

Every trial records the model actually served, not just the one requested, so the two can be told apart afterwards. Every loop carries a served-on date, and it is an invariant that one date covers the whole loop — a loop measured across two days is two experiments wearing one label.

This is why results are framed as measurements of specific endpoints at specific times rather than as verdicts. It also means a re-run months later is a new measurement rather than a confirmation, and a change between them may be the model, the routing, or the provider mix.

A result is what that endpoint did on that date.

Every figure here describes something measured and committed. See the measurements · read the method

Stop reporting a hallucination rate. Report what a wrong answer costs you.

5% is fine for a first draft and unacceptable for a dosage. The rate alone is not a risk statement.

The boring rows are load-bearing

Seven of the ten modes in our set are controls, and their flat lines are the finding.

We accepted 2,900 failures without reading them — and said so in the data

A gate everyone routes around isn't a gate. So we named the shortcut instead of pretending.

The model you tested is not the model you deployed

What we record instead

Related