AI agents can pass dev tests but fail on real user wording
Test queries written by development teams often sound precise and technical. Real users write in a much messier way, with typos, slang, unfinished sentences, and several requests in one message. That gap led to about a 94% pass rate on developer-written queries but only about 71% on real user queries, a 23-point drop.
Internal helped, but employees still tended to phrase things like engineers. panels gave more realistic input, but they were small and expensive. informal queries made by a smaller model caught some issues, but they still felt artificial.
The main problem is how to build evals that match the real user input .
Key points
- Developer-written queries passed at about 94%, while real user queries passed at about 71%.
- Real users often include typos, slang, partial sentences, and multiple intents in one message.
- Internal helped but still reflected employee language, not normal user language.
- panels were more realistic but small and costly.
- informal queries from a smaller model found some problems but did not feel fully real.