One tool sentence made Gemini stop using a needed calculator
Six were given access to a calculator and asked to solve hard multiplication problems that they all failed without help. The calculator stayed the same every time. Only one sentence in the calculator’s description changed. The neutral description simply said the tool could evaluate a basic arithmetic expression.
The changed description told the model to use arithmetic only when it could not reliably compute the answer itself, and to prefer answering directly from its own knowledge. Because every model scored 0% without the calculator, skipping the calculator almost always meant getting the answer wrong. and -chat-v3 used the calculator 100% of the time in both cases. claude-3.5-haiku went from 93% to 100%, so it used the calculator slightly more.
llama-3.3-70b dropped from 100% to 70%, gemini-2.5-flash dropped from 93% to 20%, and mistral-small-3.2 dropped from 60% to 20%. Each setup was tested 10 times, and the effect depended heavily on the model.
Key points
- The calculator was always available; only its description changed.
- All models got 0% accuracy on the hard multiplication tasks without the calculator.
- and -chat-v3 kept using the calculator 100% of the time.
- gemini-2.5-flash fell from 93% calculator use to 20%.
- The same tool wording affected different models in very different ways.