How a RAG cache can save money but return wrong answers
Putting a cache in front of RAG can cut cost and waiting time because repeated questions do not need fresh search and answer generation. In use, the cache sometimes returned wrong answers with high . One problem was stale answers: a source document changed during a session, but the cached answer still reflected the older version.
A TTL did not truly solve this, because it only allowed the answer to stay outdated for a fixed time. Another problem came from storing material, such as a summary or selected claims, instead of the full source context. A later question could need a detail that the summary left out, so the looked successful while the answer was worse than normal RAG.
A useful fix was to record the exact sources behind each cached answer and clear only the cache entries tied to a source when that source changes.
Key points
- A cache can reduce repeated and generation work.
- Cached answers can become stale when source documents change.
- A TTL limits how long stale data can remain, but it does not prevent staleness.
- summaries can drop facts that later questions need.
- Tracking source links per cache entry makes targeted in possible.