RAG quality can depend more on the query than the index
The same can give a correct answer in one product surface and make things up in another when the query is built differently. A reliable RAG flow first turns the and the latest message into one clear question that can stand alone. For harder questions, it can create 3 to 5 query variants or use HyDE, search with each one, and combine the results, but this should be limited because it adds cost.
Search should combine meaning-based with BM25 and pull a wide set of about 50 to 150 results from each. BM25 helps catch exact names, product names, and IDs that meaning-based search can miss. The results are then merged by rank, narrowed with cross-encoder rerank to about 5 to 20 candidates, and filtered with a score threshold.
The final answer should use only the supplied context, and say it does not know when the context is not enough. During , quality can improve by adding a short LLM-written note to each chunk, indexing that note in both search systems, adding likely questions or paraphrases for each chunk, splitting documents by meaning, and attaching metadata.
Key points
- Answer quality can change even when the stays the same.
- Rewrite chats into one standalone question before searching.
- Use query variants or HyDE only for harder questions to avoid extra cost on every turn.
- Combine meaning-based search with BM25 so exact names and IDs are not missed.
- Force answers to stay inside the supplied context to reduce made-up claims.