RAG mistakes often come from chunking, old indexes, and search gaps
A RAG system can give confident but wrong answers when its documents are split and searched poorly. Fixed-size splitting caused problems because small chunks removed the surrounding context needed to understand limits and details. Large chunks also caused trouble because the right section could be found, but the answer was buried inside too much unrelated text, lowering quality and raising .
A with overlap improved the results. Semantic chunking worked best, but each indexing run cost more, so it made sense only for the most important documents. A stale index created another hidden failure: documents changed, but the was not rebuilt automatically, so old information kept appearing in answers.
also struggled with exact strings such as product codes, model numbers, and specific IDs, so or was needed alongside it.
Key points
- Chunks that are too small can remove the context needed for a correct answer.
- Chunks that are too large can raise and hide the answer inside unrelated text.
- A with overlap improved .
- Semantic chunking gave the best results but cost more during indexing.
- A stale index can make an agent keep using old information after documents change.