A KV cache test flaw and a smaller 2-bit cache experiment
AI models keep earlier calculation results in a KV cache so they can generate the next words faster. This experiment argues that a common way to judge KV ization can miss the real error. Many checks run one calculation with the cache turned off, so the cache is never read.
In that setup, full , 4-bit, and 2-bit all produced the same score of 3.6416, which means the test could not see value-cache problems. The test was changed to first fill the context and then generate tokens one by one, forcing the model to read the cache. The new method rotates value vectors with a and then applies 2-bit uniform .
Keys stay on KIVI int4, and only values are changed. On the corrected test, the 2-bit value cache matched KIVI 4-bit quality to three decimal places, used about 20% less memory, and used roughly four times less memory than fp16. The result held on Llama 2 7B and TinyLlama and was reproduced on a second machine.
Key points
- A common test may miss KV ization errors because it does not actually read the cache.
- Full , 4-bit, and 2-bit all scored 3.6416 in the flawed test, showing the metric was blind to the change.
- The corrected test fills the context first and then decodes token by token.
- The method uses a plus 2-bit uniform for value-cache data.
- The result matched KIVI 4-bit quality while cutting memory by about 20%, and by about four times versus fp16.