AI memory needs tests for what it retrieves, not just answers
can look good on final answer quality while still pulling the wrong stored information. MemBench measures directly by comparing returned items with fixed belief IDs. It calculates and recall from that overlap, without a generative judge, judge prompt, equivalence rules, or a scoring layer that a provider can tune for itself.
The benchmark has 89 cases and is . Providers do not submit their own scores; the harness runs tests through each provider wrapper and publishes the results. In the shown results, tenure passed all 43 active tests and scored 1.00 mean and 1.00 mean recall.
Several other s had high recall but low , meaning they often found many relevant items while also bringing back too much wrong or extra material. speed and time also varied widely across systems.
Key points
- MemBench evaluates , not the final answer alone.
- and recall are measured by direct overlap with fixed belief IDs.
- The setup removes generative judges, judge prompts, equivalence rules, and provider-tuned scoring layers.
- The benchmark includes 89 cases and runs through provider wrappers instead of accepting provider-submitted scores.
- tenure led the shown table with 1.00 mean , 1.00 mean recall, and 9.77 ms median time.