Agent tests need to check actions, not just answers
A small tone change in a support agent’s passed all existing tests. In real use, the agent stopped calling the `lookup_order` tool when answering order-status questions.
It then gave confident answers without checking real order data. The false answers still looked fluent, helpful, and on-brand, so tests that judged only the final text missed the problem.
The weak spot was behavior, not wording. Important agent failures can include missing , calling the wrong tool, using a risky tool without a gate, , taking too many steps, or increasing and cost after a model change.
Key points
- A small change made the agent stop using its order lookup tool.
- Tests that only judged final text missed the failure because the fake answers looked good.
- Agent tests should check , selected tools, step count, and safety gates.
- Extra loops or too many steps can raise and cost after a model change.