Memory system scores 470/500 while using about 10x fewer tokens
A built on Qwen3 235B A22B Instruct 2507 scored 470 out of 500 on . Its published comparison places it first among the systems tested, while using about one-tenth as many tokens as the next-best system. Qwen3 handles storage and retrieval, while the user can choose a separate for the final response.
The main challenge was not designing the system but making smaller models behave reliably despite unexplained errors. In the developer's firsthand account, about 90% of the work went into adjusting prompts and adding safeguards. A public viewer shows the questions, prompts, and model results, while an runner lets others repeat the tests and check the scores with the official grader.
The live service uses different prompts because the benchmark prompt scaffolds were tuned heavily for Gemini.
Key points
- The system scored 470 out of 500 on .
- It used about one-tenth as many tokens as the next-best system.
- Qwen3 manages storage and retrieval, and users can select the .
- About 90% of development time went into prompt changes and safeguards.
- The runner supports repeat testing with the official grader.