ModelCensusopen-source ai reliability harness

Locums rotate. The chart stays.

rentable models, strategic data

Blog24 Sept 2026enterprisestrategyorchestrationvalidation

Rent the model. Own the chart.

The model is rentable. Your data is strategic. Treat the model like a locum doctor, and your data like the patient's chart.

Hospitals worked this out long ago. A locum doctor is skilled, is booked by the shift, and could be working at the hospital across town tomorrow. What makes a new locum safe on day one is the chart.

The locum — rent itThe modelsame weights as your competitorVector storecommodityAgent frameworkreplaced yearlyvs.The chart — own itMemorysurvives sessions and swapsGoverned live datatoday's numbers, scopedTraceabilitywhat it saw, what it didFailure recordthe allergy list
Everything on the left can be booked by anyone. Everything on the right is history only you have, and it goes with the patient, not the doctor.

Interchangeable, until it isn't

Every locum is qualified. They are not all the same. We gave 20 models identical cases and identical detectors:

best of 20worst of 20Constraint decay at depth0% → 93% · n=15Basis-demand evasion0% → 73%Citation resolution failure10% → 61%Cross-turn schema validity0% → 50%Date-arithmetic error0% → 40%Three flat modes0% → 8%
Failure rate per mode, best model against worst, bare condition, 15–50 probes per cell. The top three rows separate with non-overlapping intervals. The bottom row is flat across all 20, so on those modes the models really are interchangeable.
37%
of enterprises now run five or more models, because the models differ by use case, not because they are commodities (a16z CIO survey)

What goes in the chart

ComponentSuperpowerWhere it breaks
MemoryContinuity across swapsStale facts recalled with confidence
Governed live dataToday's numbersControls built for human readers
TraceabilityThe incident reviewShows one incident, not a rate
Fast retrievalAgent loops stay usableFast retrieval of the wrong passage
Failure recordMakes a swap provableSlow to build; rots unless re-run
The first four are the history. The fifth is the allergy list, and it is the one most charts leave blank.
THE CHARTRENT ITYOURS, BUT DISPOSABLETHE LOCUMFailure recordVector store, frameworkLineage & access policyMemory with provenancePrompts tuned to one vendorModel weightsonly youevery competitorWho else has itSurvives a model swaprebuilt on swapcarries over
The trap sits bottom-left. Prompts tuned to one vendor are yours and feel strategic, but they are the previous doctor's handwriting, and you rewrite them on every swap.

You are already swapping locums

Upgraded, same vendor66%No change23%Switched provider11%
What teams did with their model over one year (Menlo Ventures). The board debates the 11%. The 66% is also a new doctor on the ward, and it never goes through your change board.
Upgrade shipsRun the recordCompare modesPromote or hold
The handover. Without the record, the middle two steps are a vibe check.

The hard part

190 model pairs, nine modesIndistinguishable on every mode82%Separate on at least one18%
Our own census, at 15–50 probes per cell. For most pairs we cannot say whether a swap was safe. That is not a flaw in the method; it is what the claim costs. A generic benchmark can't tell you, and a thicker chart on your own cases can.
Book the model by the shift. Keep the chart forever.

When your model last changed, what did the new one get to read before its first shift?

Sources

Every figure here describes something measured and committed. See the measurements · read the method

Related

Open or closed is a label, not a risk rating
Open models trail the frontier by about four months and are showing up everywhere. I split my 20-model census by weights. The label barely predicted failure. The model did.
Every role on the chart has a next chapter
Gartner's new org chart merges twelve roles into six, and six into three teams. It reads like a list of cuts. It's a map: for each role, what AI takes, what stays human, and the first skill to learn.
My prompt pack didn't make the decks better
I wrote a prompt that guards against five ways AI gets spreadsheet numbers wrong, and tested it on 120 decks built from messy exports. It didn't help. Two things did.