A silent filter made hybrid search run vector-only for months
A customer-support RAG system stored uploaded documents, scraped web pages, and resolved support conversations using one and one pgvector table. It combined with BM25 through RRF, then reranked the results. A minimum rule was mistakenly applied after the results were combined.
Items found only by BM25 had no vector score, so the rule silently removed all of them. The supposed therefore operated as vector-only search for months, without errors or degraded-mode logs. Search became worse for exact product names, codes, fixed phrases, and terms written in a different language.
The fix was to apply the minimum score only to the vector-search branch and allow BM25-only results to remain during fusion. were added, along with customer-specific sets of known questions and answers that include trap questions the available knowledge cannot answer and the system should decline.
Key points
- Apply a vector cutoff only to the vector-search branch, not to every fused result.
- Keep BM25-only matches because they do not have vector scores.
- Test exact names, codes, fixed phrases, and cross-language queries.
- Add that prove both parts of contribute results.
- Use customer-specific test sets with questions the agent should refuse to answer.