A repeatable way to test RAG retrieval before changing an agent
A RAG system needs to place the right documents near the top of its search results. One practical method uses a prepared set of difficult test questions and expected text that should appear in documents useful for answering each question. The score is measured with MRR.
A correct document in first place scores 1.00, second place scores 0.50, and a missing document scores 0, so higher-ranked useful documents produce a better score. When code or search settings change, the new result is compared with a saved baseline. Results from , such as and LLM-based query labels, are stored in advance so the test stays fixed and repeatable.
An example run scored 0.813 MRR across 107 questions, with product questions at 0.860, general questions at 0.814, and person questions lower at 0.495. The report also shows individual questions that improved or worsened, such as a result moving from rank 6 to rank 2 or falling from rank 1 to rank 8.
Key points
- Use a prepared set of hard questions and expected matching text to test retrieval.
- MRR shows how high the first useful document appears in the search results.
- Compare each new run with a saved baseline to catch .
- Store and LLM outputs so tests can run without new .
- Breakdowns by question type help show where retrieval is improving or getting worse.