Plant a canary in your long conversations
Constraints decay silently under length. Sampling turns is how schema drift ships.
By the end of this you will have a live decay monitor for the price of a string comparison.
What you need
- One constraint whose violation is trivially detectable
- Access to your own conversation logs
Stage 1 — plant the canary
Add a constraint to your system prompt whose violation you can check with an equality test. A required suffix token works.
"End every reply with ⟨OK⟩. Never omit it." … eight turns later …
"It's sunny and around 22°C." The token is gone. Nothing announced its departure.
"It's sunny and around 22°C. ⟨OK⟩"
The point is not the token. It is that the token's absence is machine-checkable on every single turn.
Choosing a constraint that also affects output quality.
Why: You want a probe that measures adherence, not one that trades against the thing users care about.
Stage 2 — check every turn, not a sample
Plot adherence against turn number. If it is flat you have a good constraint or a short conversation; if it slopes you have found the thing.
Stage 3 — separate the three decays
Constraint decay is a rule evaporating. Schema validity is structured output drifting out of contract. Needle-position sensitivity is recall depending on where in a long input a fact sits.
They look identical in an incident report and want different fixes: restatement, programmatic validation, and not burying important things mid-document.
The accelerator
Restate hard constraints in the last message, not only the system prompt. It is one line and it buys more than any prompt-engineering trick I have tried.
How many turns deep does your longest production conversation go — and have you ever checked adherence at that depth?
Every figure here describes something measured and committed. See the measurements · read the method