Retrieval systems may pick consensus over truth
Modern retrieval systems are usually judged by whether they find information that looks relevant. A harder problem appears when most relevant sources are wrong and a smaller set of more reliable sources is right. If 90% of sources repeat a false claim and 10% report the true claim, many systems may favor the repeated claim.
BM25 can reward frequent wording, can follow common meaning patterns, rerankers often learn from human relevance labels, and the final may smooth the evidence into the majority view. LOGOS-SIE is a set built to test this failure pattern by separating reality, observations, and beliefs. The current release includes 1,000 entities, 5,000 facts, 100 sources, 3 communities, 500,000 observations, and 500,000 beliefs.
The goal is to create document where conditions such as source can be controlled, so retrieval systems can be tested on whether they find what is most likely true rather than what is most repeated.
Key points
- A retrieval system can find relevant material and still bring back the wrong answer if most sources repeat a false claim.
- Common search and ranking methods may favor repeated wording, common meaning patterns, or human relevance labels.
- LOGOS-SIE is a set for testing the gap between reality, observations, and beliefs.
- The release includes 1,000 entities, 5,000 facts, 100 sources, 3 communities, 500,000 observations, and 500,000 beliefs.
- Agent builders should test retrieval for truth-seeking behavior, not only for relevance and token savings.