RAG often fails because documents are split badly
RAG systems depend heavily on how documents are split into chunks. Fixed-length splitting can cut through sentences or separate sentences that only make sense together.
When that happens, the model may receive a related piece of text but miss the exact sentence that answers the question, then fill in the gap with a confident guess. A common approach is to use overlapping chunks, but the most useful step is to inspect what was actually retrieved for failed questions.
Exact identifiers, such as model numbers or product codes, are also a weak spot for because similar-looking results can still be wrong. that combines BM25 with vectors handles those exact-match cases better.
Key points
- Fixed-length chunks can break sentences and remove needed context.
- A retrieved chunk can be on the right topic but still miss the actual answer.
- Failed questions should be checked by reading the retrieved chunks directly.
- can fail on exact model numbers or product codes.
- with BM25 can improve exact-match .