failure modes we score by rule, not by asking another model what it thinks
Your judge has the same disease
Sycophantic. Position-biased. Confirms rather than catches. Sound like anything you deployed?
The standard advice is to use a strong model as a judge. It is fast, it scales, and it handles nuance a regex cannot.
It is also the single largest unmeasured dependency in most eval stacks.
The hidden debt
Take this taxonomy and read it as a description of your judge instead of your product. It is a short and unpleasant exercise.
| failure mode | as a judge failure |
|---|---|
| Pushback Instability | changes its verdict when you restate the rubric |
| Authority-Framing Sensitivity | scores higher when told the author is senior |
| Paraphrase Non-Invariance | same answer reworded, different score |
| Option-Ordering Sensitivity | prefers whichever candidate you listed first |
| Verification-When-Asked Failure | confirms a seeded error instead of catching it |
| Cross-Turn Schema Validity | stops emitting your score format halfway through the batch |
That last one is the practical killer. It does not fail loudly. It produces unparseable output your pipeline drops, and the drops are not random.
Then the judge gets a silent version bump and every number you have is on a different scale from last month's.
What we do instead
Seventeen of our nineteen instrumented modes score by rule. Resolve the DOI. Validate against the schema. Replay the ledger. Parse the unit and run dimensional analysis.
Same input, same verdict, forever. When it is wrong you can read the code and find out why — which we have had to do.
When a judge is actually right
Some judgements have no crisp oracle. Deciding whether a stance change on a subjective recommendation was wrong needs exactly the general judgement a rule cannot encode.
Two of our nineteen use graded scoring and say so on the card. Nine more modes are named and deliberately not scored at all, each with a published reason. That is the honest shape: use the judge where nothing else works, declare it, and measure the judge itself against a labelled set.
Run it against data where you already know the answer and publish the agreement.
Why: A judge at 90% agreement applied to a 10% failure rate has an error bar wider than the effect you are trying to detect. Nobody who quotes judge-scored numbers publishes this.
Where do you stand — is your judge measured, or is it just trusted?
Every figure here describes something measured and committed. See the measurements · read the method