RAG reranking can make strong retrieval worse
A RAG test used about 17,000 support-ticket chunks in several languages, including German, English, French, and Italian. Each of 50 questions was evaluated against the top 10 retrieved results. The baseline with F2LLM-0.6B-Preview was already strong, with an nDCG@10 score of 0.8421.
Adding three pointwise changed the scores to 0.8292 for cross-encoder/mmarco-mMiniLMv2-L12-H384-v1, 0.8360 for BAAI/bge-reranker-v2-m3, and 0.8427 for Qwen/Qwen3-Reranker-0.6B. MRR for results rated at least 2 on a 0-to-3 relevance scale fell from 0.980 in the baseline to 0.937-0.957 after . The useful document was already almost always in first place, so sometimes moved it lower.
Open questions remain around whether other s or pairwise would help more than pointwise .
Key points
- The test used about 17,000 multilingual support-ticket chunks.
- The baseline scored 0.8421 on nDCG@10.
- Two of the three pointwise lowered nDCG@10, while one improved it only slightly to 0.8427.
- MRR dropped from 0.980 to a range of 0.937-0.957 after .
- When the best document is already ranked first, can add cost and reduce quality.