The component that produces the numbers has no write access to the site. That separation is the whole design.
harness architecture
The runner that cannot publish
The architecture decisions I'd defend, and the one I'd change.
Most eval harnesses are a script and a spreadsheet, which works until someone asks where a number came from. Here is the structure we ended up with and what each piece costs.
The four levels
Where it shines: every number has a containing scope, so "where did this come from" is a traversal rather than an investigation.
Where it breaks: it is rigid. A question that does not fit the probe-card-loop shape — anything comparing across sets, for instance — needs work outside the model, and we have written that work twice.
The injected sink
What it does: the runner writes trials through an interface rather than importing the persistence layer. The evals package has no database dependency at all.
Where it shines: somebody can lift the detectors and case format into their own harness without inheriting our storage decisions. That was the point.
Where it breaks: one more indirection to follow when debugging, and the sink implementation is the least-tested code we have because it is boring.
Manifests, not a database
What it does: the site reads committed JSON rather than querying at request time.
Where it shines: the deployed artefact and the repository are the same thing. `git log` is the publication history and a diff shows exactly what changed about a claim.
Where it breaks: builds get slower as the corpus grows. We are at 434 static pages and it is fine; at ten times that it would not be.
This one was a correction, not a design. When pages queried live state, a row edited in the database changed what the site claimed with no commit, no review, and no way to tell afterwards.
My default setup
Keep: manifests in git, and a runner that writes `status: review` and cannot clear it. Two of our fourteen publish invariants have no command that satisfies them, which is what stops the pipeline publishing itself.
Change: I would build the cross-set comparison into the model from the start rather than bolting it on. That is the piece we have rewritten most.
A pipeline that can mark its own output reviewed is a pipeline with no review gate, wearing one as decoration.
Where does your eval data actually live — and can you diff what a number said last month?
Every figure here describes something measured and committed. See the measurements · read the method