KyroBench tests whether agent context is safe to use
KyroBench is a benchmark for checking whether the context given to an AI agent is safe to use before the agent answers or acts. It goes beyond finding text that looks similar to the question. It checks whether old information loses to newer valid information, deleted memory stays deleted, another customer’s data stays out, and misleading near-duplicates are rejected, and returned evidence can be verified.
The test cases cover everyday production settings such as , billing exceptions, legal contracts, incident runbooks, CRM memory, synthetic healthcare records, and traces. The current official run covers 6 tracks, 36,864 scored checks, 12,288 retrievals per run, and a 1,200 token . The official score is treated as a certification gate, while retrieval signal and latency are shown only as diagnostics.
KyroDB, Graphiti/Zep, Qdrant, and Mem0 all scored 0 on official certification. KyroDB had the strongest retrieval signal at 79.1 and complete proof metadata, but missed the hidden semantic and freshness gates. Graphiti/Zep had a 71.6 retrieval signal and the fastest p95 latency at 168 ms, but lacked the proof surface needed for certification; Qdrant acted as a raw vector-store baseline with a 53.3 retrieval signal; Mem0 blocked many strict requests as stale and produced almost no completed retrieval signal.
Key points
- KyroBench checks whether agent context is current, in scope, clean, provable, and small enough for the .
- All four tested systems received 0 official certification in the current run.
- High retrieval signal did not guarantee certification because similar text can still be stale, unsafe, or unverifiable.
- The benchmark uses a 1,200 token , so it also tests whether key evidence survives compression.
- The biggest practical risks are deleted memory returning, tenant data leaking, weak sources being trusted, and proof going missing.