Agent memory may not need a vector database alone
In a test of memory for agents that handle many conversation sessions, putting everything into one did not clearly beat simpler search. The benchmark used LoCoMo, a dataset of real conversations with about 5,900 turns and 1,531 answerable questions. Six retrieval methods were compared: keeping only the most recent turns, BM25, two embedders, and BM25 mixed with each embedder using RRF.
The recent-turns method scored about 0.024, which means it retrieved almost nothing useful, and it lost across all 10 conversations. The needed fact was often in an older session, outside the recent window. A single with a strong embedder was only roughly tied with BM25.
mxbai-embed-large scored 0.526, while BM25 scored 0.552, and the difference was not statistically clear. The practical result is that a cheap, hybrid of BM25 plus an embedder may be a better default than using a by itself.
Key points
- Recent-turn memory performed very poorly for .
- A single did not clearly outperform BM25.
- mxbai-embed-large scored 0.526, while BM25 scored 0.552.
- The cheaper BM25 plus embedder hybrid is presented as the stronger practical option.
- Agent builders should test instead of assuming recent context or is enough.