Agentic RAG can fail quietly when retrieval drifts
An system had been running in production for about six months with four steps: retrieval, reranking, answer writing, and output checks. A few weeks earlier, the retrieval step began pulling document chunks that were close to the topic but not actually correct. The still produced confident answers, so the problem was not obvious from the final text.
The issue surfaced through a support ticket from a user who felt the information seemed old. The cause was a document update that slightly changed the embedding , which made the top-k search results bring back nearby but wrong chunks. Weekly unit tests, built around known problems, and small manual checks after deployment did not catch this slow drift.
helped track prompt versions and answer changes, but it did not watch the retrieval step or show that the vector layer had shifted.
Key points
- The pipeline had been live for about six months.
- Retrieval started returning close-looking but wrong document chunks.
- The turned those bad inputs into confident answers.
- The root cause was a document update that shifted the embedding .
- Prompt and answer tracking did not reveal the vector layer problem.