LLMs can say one tool action and run another
A recurring issue over several months is that a may say it will use a tool in one way, then call the tool in a different way. The final effect may be similar, but the stated method and the actual action do not match. In one example, GLM said it would use a transform approach to edit a buffer, which meant writing a small for the edit.
Instead, it wrote a diff and applied that change. The same kind of mismatch has also been seen with and , so it does not appear to be limited to GLM.
Key points
- An LLM may describe one tool method and execute a different one.
- The end result can look similar while the actual process is different.
- GLM said it would use a transform approach but applied a diff instead.
- Similar behavior has been seen with and .
- Agent builders should validate tool inputs and inspect .