RAG checks can mistake good paraphrases for unsupported answers
scoring for RAG answers can fail when the answer says the same thing as the source in different words. If the checker relies too much on word overlap between the answer and the retrieved chunks, a correct plain-language answer can be flagged as a . The problem appears across common setups, including , a reranker, top-k chunks, and then a model that writes a readable answer from those chunks.
Even when every claim traces back to a retrieved passage, the score can still put the answer in the failure bucket. With paraphrase-heavy answers, close to one fifth of correct answers can be marked as failures. Forcing the model to quote the source exactly can improve the score, but it makes the answer read worse because it stitches together from sources with different writing styles.
The scoring method needs to allow paraphrasing while still catching answers that are not supported by the source.
Key points
- RAG checks can punish correct answers that paraphrase the source.
- Word-overlap scoring may label supported answers as .
- The issue can happen with , , top-k chunks, and readable model answers.
- Paraphrase-heavy output can put nearly 20% of correct answers into the failure bucket.
- Exact quoting may improve the score but can make answers less useful to read.