Why our panel keeps gpt-3.5-turbo
It isn't there to lose. It's the control for 'did this actually improve'.
A panel of only current frontier models answers one question: which of these is best right now. Short shelf life, crowded field.
We group by the axis each model is there to test.
Frontier — the ceiling
Where it shines: tells you what is achievable today. If a mode is still failing here, it is not a scale problem.
Where it breaks: expensive and slow. One frontier model was 32% of our entire census bill and took three hours.
Small and cheap — the deployment reality
Where it shines: this is what most products actually run. Rates here are the ones your users will meet.
Where it breaks: easy to mistake for the frontier when a mode happens to be easy.
Legacy — the control
Where it shines: it is the only way to answer "did this improve". gpt-3.5-turbo and claude-3-haiku are not competing; they are the baseline that makes a trend claim possible.
Where it breaks: people read the panel as a ranking and ask why old models are in it. Every single time.
Open weights, and reasoning
Open weights answer "does this failure depend on the deployment or the weights". Reasoning models answer "does thinking before answering change what a document does to it".
Where reasoning models break: cost and time. They generate far more tokens per question, which means more exposure to rate limits and dropout — and dropout lands on the hardest cases.
My default
For a new mode: two frontier, two small, one legacy. Five models tells you whether the failure is capability or behaviour, which is the only thing you need before deciding to instrument it properly.
The finding that justifies the whole spread: some modes are flat across the entire panel, old and new, large and small. That flatness is what licenses reading movement elsewhere as real.
Does your eval panel include a model you would never ship? It probably should.
Every figure here describes something measured and committed. See the measurements · read the method