Feedback helps a RAG search system forget outdated answers
A pipeline was modified to learn from feedback. Each retrieved text chunk gets that are updated according to whether the chunk helped produce useful past answers. then changes the order of the candidates for every new question.
In a where the correct answer changed over time, a fixed search system remained at about 42% accuracy because it kept choosing chunks that had been correct before. Adding decay reduced the influence of old feedback, allowed the system to drop stale preferences, and raised accuracy to about 73%. The method cannot help when the initial search fails to include the correct chunk among its top candidates.
It also needs enough feedback for each group of similar questions; before that feedback accumulates, it behaves like ordinary . deliberately tests uncertain choices, so its results may look slightly noisy when the correct answer remains stable. The code and experiment notes are available as .
Key points
- record whether each text chunk helped previous answers.
- reorders the search candidates for each new question.
- With decay, accuracy recovered from about 42% to 73% after the correct answers changed.
- The method cannot recover a correct chunk that is missing from the initial top candidates.
- Learning is limited until enough feedback builds up for similar questions.