RAG failed in production because search quality mattered most
A RAG demo worked well at first, but answer quality dropped when real users started using it. Plain was not enough because text chunks that looked similar in meaning were often wrong on the facts. , combining BM25 with dense search, plus a reranking step improved results more than switching models.
Chunking had a bigger impact than model choice. Fixed-size chunks damaged tables and code, while splitting documents by their structure fixed many of those problems. Without , there was no reliable way to know whether changes helped.
A golden dataset made it possible to measure after every change. About half of the changes that seemed better were actually , and most gains came from retrieval engineering rather than prompt changes.
Key points
- alone can pick chunks that are close in meaning but wrong in fact.
- and reranking improved quality more than changing the model.
- Fixed-size chunking can break tables and code.
- A golden dataset helped measure after each change.
- Most useful gains came from retrieval engineering, not prompt tweaks.