reported gap between agentic benchmark scores and real-world deployment performance
The 37-point drop nobody budgets for
Lab benchmark scores and production performance are not the same number, and the gap has a size.
Enterprise agentic systems have been reported showing a 37% gap between benchmark scores and deployment performance, with roughly 50× cost variation between systems reaching similar accuracy.
Neither number is a scandal. Both are entirely predictable, and that is the problem.
Expectation versus reality
The expectation is that a benchmark measures capability, capability transfers, and a system scoring 85 in the lab scores roughly 85 in production. Every procurement deck ever written assumes this.
The reality is that a benchmark measures capability under conditions chosen to isolate it. Production supplies the conditions the benchmark removed: ambiguous phrasing, irrelevant context, a tool that times out, a user who pushes back. Those are not noise around the capability. They are separate failure modes, and none of them were scored.
Diagnosis
MMLU and MMLU-Pro sit above 88% for frontier models, which makes differences at the top statistically meaningless.
Why: When every model scores the same, the score stops carrying information — but it keeps appearing in decks, because it is the number everyone recognises.
It is made of specific, nameable failures: prompt-sensitivity, sycophancy under pushback, tool-return groundedness, instruction decay over long sessions.
Why: A composed gap can be decomposed and each part measured. An 'AI is unreliable in production' gap can only be worried about.
50× cost differences between systems reaching similar accuracy means accuracy was never the operative variable.
Why: If two systems agree on the answer and disagree by 50× on the bill, the thing you should be measuring is whatever produced the 50×.
What we do now
Stop treating the gap as a discount to apply to a benchmark score, and start treating it as a set of measurements you have not taken. Our own panel exists because of this: the same twenty models, the same cases, under four conditions chosen to separate capability from the conditions around it.
The useful consequence is that the gap becomes actionable. A 37-point drop is a mood. A 13% prompt-sensitivity rate with an interval of 7 to 21 is a work item.
The gap between the benchmark and production is not degradation. It is the arrival of the failure modes the benchmark was designed to exclude.
When your system underperforms its evaluation, can anyone in the building name which failure mode accounts for it?
Every figure here describes something measured and committed. See the measurements · read the method