Wrong RAG answers may come from bad retrieval, not the model
A team’s RAG system answered questions using its own documents, but many answers were confidently wrong. The first reaction was to blame the model for making things up, so the team changed the prompt, lowered the , and tried other models.
Those changes either felt temporary or did not improve the answers much. Looking at the traces for each bad answer showed what context had been sent into the model.
The model often received poor or irrelevant context, so it produced answers based on bad input. Separating the problem into and generation made the work clearer: first check whether the right context was found, then check whether the model used that context correctly.
Key points
- A RAG setup over internal documents kept producing confident but wrong answers.
- Prompt changes, lower , and model swaps did not solve the issue well.
- Traces showed that poor context was being sent to the model.
- The team found it easier to debug by scoring and generation separately.