The four eval layers, and which one you're actually missing
Capability benchmarks. Failure-mode census. Your app evals. Production traces. Most teams have two.
Most arguments about benchmarks are really arguments about which layer someone thought they were discussing. There are four, they answer different questions, and nearly every team is missing the same one.
Layer 1 — capability benchmarks
What it does: ranks models on broad task performance. MMLU-shaped.
Where it shines: procurement. If you need to shortlist three models from thirty, this is the fastest filter that exists.
Where it breaks: it pools unrelated failures into one score. A model that is strong on average can be the worst on the specific mode that will hurt your product, and the ranking cannot show you that.
Layer 2 — the failure-mode census
What it does: per-mode failure rates with intervals, across a fixed public panel, under controlled conditions.
Where it shines: naming. You cannot track a regression in a failure you have not named, and this layer supplies the vocabulary everything downstream debugs in.
Where it breaks: it cannot tell you whether the model does your job. The panel is fixed and public, so there is no private-instance mode — deliberately, because numbers nobody outside your company can check are worth less than they appear.
Layer 3 — your application evals
What it does: measures whether the system does the specific task you ship.
Where it shines: it is the only layer that knows your users, your data and your definition of correct.
Where it breaks: almost every one I have seen is two-armed and has no control, so it cannot separate a real improvement from a prompt-length effect. It also inherits whatever vocabulary the team happened to invent, which is usually 'hallucination' doing the work of four distinct modes.
Layer 4 — production observability
What it does: traces, latencies, cost, and whatever your guardrails caught.
Where it shines: it is the only layer with real traffic in it.
Where it breaks: it tells you what happened, never whether it was avoidable. Without a named taxonomy upstream, every incident is bespoke.
My default setup
For production: layers 3 and 4, always, and layer 2's controls copied into layer 3. That is the transferable part — a length-matched irrelevant control, intervals at effective sample size, a tag split separating traps from answerable cases.
For model selection: layer 1 to shortlist, layer 2 to decide. Never layer 1 alone.
Copy the controls, not the numbers. Our rates describe specific endpoints on specific dates. The method survives the endpoints.
Which layer is missing in your stack — and is that a decision or an accident?
Every figure here describes something measured and committed. See the measurements · read the method