Local Qwen retrieval beats some scores but is far slower than FAISS
ClawIndex is a one-person retrieval project that searches an index with a reasoning step instead of relying only on text . It does not use or a , and the full test ran on a Qwen model, so the data stayed on the local machine. The covered 700 questions: 500 from HotpotQA, 100 from BEIR ArguAna, and 100 from BEIR SciDocs. ClawIndex had the best NDCG@10 score across all five dataset splits, including 0.934 on the full HotpotQA test set used here.
Its clearest lead was on multi-hop bridge questions, where it scored 0.920 versus 0.837 for FAISS. In 51 of the 500 HotpotQA questions, it used a fallback path, but each case still produced a valid result with a trace. The weaknesses are important. FAISS beat it on MRR@10 for both BEIR datasets, and the SciDocs gap of 0.975 versus 0.972 is too small to treat as a firm win with only 100 examples.
The HotpotQA setup used the distractor setting, not the fullwiki setting, so comparisons with published systems such as PRISM may not be fully fair. Speed is the biggest drawback: ClawIndex took about 27 seconds per query, while FAISS took about 7 milliseconds.
Key points
- ClawIndex uses reasoning over an index instead of and a .
- All tests ran locally on Qwen, with no data leaving the machine.
- The used 700 questions from HotpotQA, BEIR ArguAna, and BEIR SciDocs.
- ClawIndex led on NDCG@10 across all tested splits and did especially well on multi-hop questions.
- The main practical problem is speed: about 27 seconds per query versus about 7 milliseconds for FAISS.