ModelCensusopen-source ai reliability harness

An endpoint is not a version. The weights behind it can change without the string in your config changing at all.

silent model drift

Blog5 Aug 2026conceptmethodenterprisestrategy

The model you tested is not the model you deployed

A model id used to name weights. It now names a routing decision — and most AI assurance has not noticed.

Here is a weak signal most AI assurance programmes are ignoring: the model id in your config no longer identifies a model.

It identifies a routing decision. And that decision changes underneath you.

Your requestmodel idRouterpicks a providerProviderquantization variesResponsewhat you measured
Three of these four can change without the id changing.

The forces

  • Routers spread one id across several providers
  • Providers serve the same weights at different quantization and configuration
  • The weights behind an alias get updated, and the alias stays the same

Old way, new way

Model as artefactTest at launchPin the nameAssume it holdsvs.Model as serviceRecord what was servedDate every resultRe-measure on a schedule
The launch eval becomes obsolete. The recurring one becomes the asset.

What we do: every trial records the model actually served, not just the one requested. Every loop carries a served-on date, and it is an invariant that one date covers the whole loop. A loop measured across two days is two experiments wearing one label.

How this plays out

nowResults quoted against a model name.0–6 moUnexplained regressions with no config change.6–12 moContracts start naming served version and date.12–18 moScheduled re-measurement replaces the launch eval.
A prediction, not a measurement. The first row is already true.

How to prepare

Log the served model on every call

Most APIs return it. Almost nobody stores it.

Put a date on every result you quote

A result is a measurement of an endpoint on a date — not a permanent property of a model.

Re-run a fixed probe set monthly

Small, frozen, cheap. Its only job is to tell you whether the thing behind the id moved.

Why: A re-run months later is a new measurement, not a confirmation. Treat a change as a finding.

A result is what that endpoint did on that date.

If your provider changed what sits behind your model id tonight, which of your dashboards would show it?

Every figure here describes something measured and committed. See the measurements · read the method

Related

Open or closed is a label, not a risk rating
Open models trail the frontier by about four months and are showing up everywhere. I split my 20-model census by weights. The label barely predicted failure. The model did.
Every role on the chart has a next chapter
Gartner's new org chart merges twelve roles into six, and six into three teams. It reads like a list of cuts. It's a map: for each role, what AI takes, what stays human, and the first skill to learn.
My prompt pack didn't make the decks better
I wrote a prompt that guards against five ways AI gets spreadsheet numbers wrong, and tested it on 120 decks built from messy exports. It didn't help. Two things did.