Choosing Benchmarks for a Research-Paper RAG System
An early research-paper system is being designed with the scholarweave/arxiv-latex dataset. The main difficulty is the volume of material and the large number of benchmarks with different goals. Some benchmarks cover general search, scientific papers, coding problems, English forums, or .
Others test , an alone, a complete pipeline, , or the quality of the generated answer. A separate is available for LLM-as-a-judge scoring. LitSearch is one possible benchmark, although it mainly used paper titles and abstracts.
Its results suggested that adding paper context did not help much, but the test only included roughly the first 512 to 2,000 tokens of each paper, depending on the . BEIR is also under consideration, but the available material gives no implementation details or results for it.
Key points
- The planned system uses the scholarweave/arxiv-latex research-paper dataset.
- Different benchmarks test , retrieval, complete pipelines, generated answers, or .
- A could provide LLM-as-a-judge scoring.
- LitSearch mainly evaluated paper titles and abstracts.
- Its paper-context test used only the first 512 to 2,000 tokens, so it did not fully test complete papers.