A local RAG test shows the hard part is trusting the LLM judge
A local RAG was run over about 26 dense technical books. The books contain formulas, tables, exact numbers, and parameter values, so copying one figure incorrectly counts as a real failure. The setup used , 128GB of memory, Ollama, and fully offline models. The had two layers: retrieval and .
Retrieval looked solved, with Recall@8 at 100% and MRR around 0.98. The uncertain part was the judge layer. The answer model was qwen3.5:122b, while the judge was llama3.3:70b-q8 from a different model family to reduce self-bias. Across 4 books and about 80 questions, the averages were about 91% correctness, 89% relevance, 60% , and a 10% rate.
The numbers are hard to reconcile because an answer should not be 91% correct if 40% of its claims are unsupported. The likely explanations are that answers add extra unsupported detail, or that the judge is too strict when it breaks answers into small claims. The judge has not yet been checked against hand-labeled answers, and the open question is whether measures like Cohen's kappa, reasoning judges, DeepSeek-R1-distill, or Llama 4 would give more trustworthy results.
Key points
- The setup tested local offline RAG over about 26 technical books.
- Retrieval was strong, with Recall@8 at 100% and MRR near 0.98.
- The judge reported about 91% correctness, 89% relevance, 60% , and 10% rate.
- High correctness and low may mean the answers add unsupported details, or the judge is too strict.
- An LLM judge should be compared with hand-labeled answers before it is trusted for .