No vendor funds this, which rules out a great deal we could otherwise have said, and is the only reason the rest is worth reading.
on independence
Nobody pays us to say this
A five-minute check for whether any AI benchmark is safe to quote in a board paper — and what passing it cost us.
Every benchmark says it is independent. By the end of this you will have a five-minute check for whether one actually is — before its number goes into your board paper.
What you need
- The benchmark's methods page or repository
- The single number you want to quote
- Five minutes
Stage 1 — was the panel fixed before the run?
Look for a model roster committed, with a date, before any results existed.
Why: A roster that can change after results can quietly lose the model that embarrassed someone.
Stage 2 — can you read the scoring?
The detector or grading code is public, or at minimum the judge prompt is.
Why: An unpublished scorer is an opinion with a leaderboard attached.
Stage 3 — does the number open?
Click from the number you want to quote to the raw output that produced it.
Why: If you cannot, nobody you show it to can either.
Stage 4 — who saw it before you did?
- No vendor sees results before publication
- No embargo, no sign-off
- Model costs paid personally — no advertising, nothing for sale
- Corrections open to anyone, published as dated errata
What passing it cost us
A fixed public panel means I cannot quietly drop a model that produces an inconvenient result. And there is no private-instance mode: running this against your deployment would produce numbers nobody outside your company could check.
Both cost adoption. Both are the point.
The accelerator
If you only have thirty seconds, ask one question: can I click from this number to the output that produced it? Most benchmark numbers in most decks fail right there.
The last benchmark number that made it into one of your decks — could anyone have clicked through to the transcript?
Every figure here describes something measured and committed. See the measurements · read the method