AI memory benchmarks may no longer show real differences
is used to judge , but its results are being questioned. The benchmark came out in late 2024 and mainly tests whether a system can answer questions from a chat transcript. That covers only a small part of what a real must do, such as keeping useful facts, finding them later, and avoiding wrong memories.
Top scores are now clustered around 90% to 95%, so the test does not clearly separate stronger systems from weaker ones. A bigger concern is that high scores with little detail can get more attention than lower scores with a clear method. For example, a fully explained 81.5% result can be overlooked beside a 95% claim that does not show how it was run.
A better setup would use a trusted group, a , and the same fixed models for everyone.
Key points
- is a benchmark for testing .
- It mainly tests question answering over chat , which is narrower than real work.
- Top results are bunched around 90% to 95%, making the ranking less useful.
- Clear 81.5% results can lose attention to vague 95% claims.
- A stronger benchmark would use a trusted owner, a , and fixed models for all teams.