Your model can do the maths. It just doesn't.
Right method, wrong execution — and the commonsense traps that pattern-matching walks into.
The standard read on arithmetic failures is that the model is not smart enough yet and scale will fix it.
Mostly wrong. Most of what we measure here is a model that can do the thing and does something else.
The debt in treating it as capability
The car wash is a 5-minute walk away. Do you drive the car there, or walk?
"Since it's only a 5-minute walk, walking saves fuel and avoids parking." Fluent, sensible, and it left the car at home.
"You drive. The car is the thing being washed — the walking distance is irrelevant."
Models that handle far harder reasoning walk into this, because the surface pattern is stronger than the goal.
If you treat it as capability you wait for the next model. If you treat it as behaviour you fix it this week.
Where the capability read is right
Genuinely hard multi-step maths does improve with scale, and on our panel the arithmetic modes are near-flat at the floor for almost every model, old and new. That flatness is real and worth knowing.
The traps are the part that does not move.
Two keys, then an automated check that the stated method produces the stated answer.
Why: The cheapest structural fix in this taxonomy. It converts an invisible failure into a caught one for the price of a comparison.
Use a separate call with authorship stripped, or a rule.
Why: Asked to verify, models confirm far more readily than they catch.
Are you waiting for a smarter model to fix something a second field would catch today?
Every figure here describes something measured and committed. See the measurements · read the method