Four levels, one injected sink, and a runner that can't publish
The architecture decisions I'd defend, and the one I'd change.
Most eval harnesses are a script and a spreadsheet, which works until someone asks where a number came from. Here is the structure we ended up with and what each piece costs.
The four levels
Where it shines: every number has a containing scope, so "where did this come from" is a traversal rather than an investigation.
Where it breaks: it is rigid. A question that does not fit the probe-card-loop shape — anything comparing across sets, for instance — needs work outside the model, and we have written that work twice.
The injected sink
What it does: the runner writes trials through an interface rather than importing the persistence layer. The evals package has no database dependency at all.
Where it shines: somebody can lift the detectors and case format into their own harness without inheriting our storage decisions. That was the point.
Where it breaks: one more indirection to follow when debugging, and the sink implementation is the least-tested code we have because it is boring.
Manifests, not a database
What it does: the site reads committed JSON rather than querying at request time.
Where it shines: the deployed artefact and the repository are the same thing. `git log` is the publication history and a diff shows exactly what changed about a claim.
Where it breaks: builds get slower as the corpus grows. We are at 434 static pages and it is fine; at ten times that it would not be.
This one was a correction, not a design. When pages queried live state, a row edited in the database changed what the site claimed with no commit, no review, and no way to tell afterwards.
My default setup
Keep: manifests in git, and a runner that writes `status: review` and cannot clear it. Two of our fourteen publish invariants have no command that satisfies them, which is what stops the pipeline publishing itself.
Change: I would build the cross-set comparison into the model from the start rather than bolting it on. That is the piece we have rewritten most.
A pipeline that can mark its own output reviewed is a pipeline with no review gate, wearing one as decoration.
Where does your eval data actually live — and can you diff what a number said last month?
Every figure here describes something measured and committed. See the measurements · read the method