A RAG bot can miss an answer that was in the docs all along
A chatbot can say it does not know the answer even when the answer is already in its . The answer was in `sla-policy.md` from the beginning: Priority 1 incidents had a one-hour response SLA and a four-hour SLA. The failure happened because the user asked with short forms like “SLA” and “P1,” while the document used longer written-out wording.
The embedder did not expand those acronyms, so the query and the correct text chunk had a score of only 0.41. The correct chunk ranked 47th, far below the top 5 chunks passed to the model. The model was not broken; it was missing the right context.
A practical fix is to test recall@5 with 12 to 15 questions written the way real users ask them, including acronyms, code names, and s, and run that test in CI whenever the embedder changes.
Key points
- The answer was already indexed, but it did not appear in the top results.
- The user used “SLA” and “P1,” while the document used longer wording.
- The correct chunk ranked 47th, so it was not sent to the model when only the top 5 chunks were used.
- A small recall@5 test set can catch this kind of failure.
- CI can warn the team when changing the embedder makes search quality worse.