Small models’ confidence is weak for agent abstention
Using a model’s own to decide whether an agent should answer or hold back is unreliable for small and mid-sized models in this test. The setup used randomly generated multi-step integer arithmetic, so the answers could be graded exactly and were not based on memorized examples. Each model returned both an answer and a from 0 to 100, then was compared with whether the answer was actually correct.
A score of 0.5 means gives no useful signal, while 1.0 means it perfectly separates right answers from wrong ones. qwen2.5:7b scored 0.50, and qwen3-coder:30b scored 0.54, so their was close to useless for deciding when to abstain. Both models were also strongly overconfident, with far above real accuracy.
glm-5.2 scored 0.73, and scored 0.90, with Claude also showing only a small gap between and real accuracy.
Key points
- Small and mid-sized models did not reliably know when their own answers were wrong.
- qwen2.5:7b scored 0.50, which means its gave almost no signal.
- qwen3-coder:30b scored 0.54, only slightly better than chance.
- glm-5.2 scored 0.73, while scored 0.90.
- Cost-saving agent designs should add separate checks instead of trusting small models’ alone.