Our chatbot works great for 5 turns, then loses the plot — evals missed it
The chatbot performs well for roughly the first five turns of a conversation, but after that its coherence breaks down. It forgets the user mentioned earlier, contradicts something it said a few messages before, or answers the latest question while ignoring context that was still relevant. It's not a total breakdown, just a gradual drift that makes the conversation feel off.
The team had built out using and consistently passed their quality thresholds, yet this problem was never caught. The reason: almost every test case was a single prompt-and-response pair, checking only whether the model answered one question correctly. What was actually needed was a way to test whether the model stays coherent across a full 15-20 turn conversation.
The team hasn't figured out how to design — whether to build datasets of entire conversations and score them end-to-end, or evaluate each turn individually and then aggregate the results.
Key points
- The chatbot works fine for about the first 5 turns, then coherence degrades
- Symptoms: forgetting earlier , contradicting itself, ignoring relevant prior context
- Evals on kept passing quality thresholds but missed the issue because almost all test cases were single-turn
- What's actually needed is evaluating coherence across full 15-20 turn conversations
- Still unclear whether to score entire conversations end-to-end or evaluate turns individually and aggregate