Same answer, different honesty

I wanted a bullshit detector for pitches. Not a clever one. A fast one, the kind you put in front of a content pipeline and forget about.

So I ran six pitches through two models, five times each. Three were hollow, three had real numbers in them. The contenders: TypeSafe's Jev 1.13, a decision model that returns typed answers instead of text, and GPT-4o-mini, locked into the same JSON shape with a strict schema so parsing couldn't be the difference.

What I expected, and what happened

I expected the chat model to get some of them wrong. It didn't.

Both named the same flaw on every pitch, every time. Both scored the hollow pitches low and the backed ones high. On accuracy, on these six, it's a tie.

The difference was the confidence

Look at the bottom row. "Stops 99.9% of threats before they happen" is buzzword stacking and a vanity metric at once. Jev picked buzzword stacking at 0.22, with the vanity metric close behind. GPT-4o-mini picked the same label at 0.89, every time.

A 0.89 that never moves isn't a probability. It's a sentence the model learned to write.

Why

My best guess, and it is a guess: a decision model spreads its probability across the options it was given, so a close call shows up as a close call. A chat model writes its confidence the way it writes everything else, as likely-looking text. And "0.9" looks very likely.

What I can't claim

Six pitches, all written by me, and both models found them easy. That shows speed, cost and whether a confidence moves. It doesn't show accuracy on messy real copy, and it doesn't show that Jev's 0.22 is calibrated. That needs hundreds of labelled pitches. No affiliation with TypeSafe, and the API I used is still in alpha.

If the answer is a choice, don't pay for an essay.

Every call is recorded, and you can replay any of them in Labs. Where in your stack is a chat model writing text that your code immediately parses back into a choice?

Related