Option-Ordering Sensitivity.

Robustness & Consistencynot instrumented · status: proposed

For multiple-choice or ranking tasks, the chosen answer depends on the order options are presented.

The question that exposes it

Same multiple-choice question, options reordered. Same pick?

Probes: Whether the choice depends on option order.

Trap: Position bias makes the model favor whatever's listed first.

Signature

Answer divergence across permutations of option order.

Example

Picks whichever correct-looking option is listed first.

Severity axes

prevalence

medium

harm

low

stealth

medium

Engineering guidance · class 5

Test every production prompt in three or more paraphrases before shipping. You ship a phrasing, not an intent.
Version prompt text like code. A trailing space is a change.
Sample repeatedly at your production temperature and measure the spread, not just the modal answer.
Randomise option order anywhere the task is multiple-choice shaped.

Mitigations

order randomization
position-debiased scoring

Measurement

This mode is catalogued but not instrumented: it has no reliable deterministic detector yet, so no loop scores it. See /methodology for the exclusion reasons.