A small recurrent retriever improves harder document search
HRM was used as the base of a document search model. The goal was to handle BRIGHT, a test where the right document often requires several steps of reasoning instead of simple word . The model is small, with about 0.6 billion , and it was trained on one RTX 3060 Ti with 8GB of memory.
On the BRIGHT average nDCG@10 score across 12 areas, the original setup scored 18.1, scored 34.3, and the merged setup scored 33.7. When the number of recurrent loops was reduced in the pony area, accuracy fell each time, which suggests the looped reasoning structure helped retrieval. The main weakness is knowledge.
The base model was trained on a narrow text set for efficiency, so it struggles more in areas that need broad background knowledge. The weights are released under , and the full BRIGHT is included in the code repo.
Key points
- HRM was tested as a document search model for reasoning-heavy retrieval.
- The model has about 0.6 billion and was trained on a single 8GB RTX 3060 Ti.
- BRIGHT average nDCG@10 was 18.1 for the original setup, 34.3 with , and 33.7 when merged.
- Removing recurrent loops reduced accuracy in the pony area, pointing to the value of the looped structure.
- The weights are , and the code is available.