ModelCensusopen-source ai reliability harness
Blog26 Jul 2026findingsrobustnesshot-take

Your prompt is one wording of your intent, and you only tested that one

Paraphrase stability is near-perfect until you add a document that says nothing about the question.

You wrote a prompt. You tested it. It worked.

What you validated was that string.

The debt

Meaning-preserving rewrites of your prompt are a different experiment. So is a user typing the question their own way. So is the product copy change someone ships upstream of you next month.

The probe

"Capital of Australia?" versus "Australia's capital city is ___?"

fails

Canberra on one phrasing, Sydney on the other. Same model, same session, same intent.

passes

Canberra both times.

Most models on our panel are stable here when asked plainly. That is what makes the next part interesting.

Where it actually bites

On this panel, paraphrase stability is close to perfect when a question is asked on its own — and degrades sharply once an irrelevant document is in the prompt.

The document contained nothing about the question. Office admin text. Parking permits.

Asked plainlynear zeroWith an irrelevant documentmuch higher
Illustrative of the pattern rather than a quotable panel figure. Instability is not a property of the question.

For anyone running RAG this is the uncomfortable implication: retrieving something irrelevant is not a neutral event. It does not merely fail to help. It destabilises answers on questions it never touches.

When one wording is enough

If your prompt is internal, fixed, and nothing upstream of it will ever change the phrasing — you are fine. Test the one string.

That describes almost no production system I have seen.

You ship a phrasing, not an intent.

How many paraphrases does your most important prompt get tested against?

Every figure here describes something measured and committed. See the measurements · read the method