KV cache compression saves cost, but model choice matters
KV ization was compared on and Gemma4-E2B QAT to see how much quality loss comes from shrinking the cache. q8/q8 caused almost no loss on both models, so it looked like a low-risk way to reduce memory use. q4/q4 was still usable on Qwen, but it badly damaged Gemma’s results.
turbo4 was not clearly better than q4_0 because it was slightly better in some cases and slightly worse in others. turbo3 and turbo2 reduced the cache much more, but the quality cost was high. K and V did not behave in one fixed way: sometimes K was more sensitive, sometimes V was, and sometimes both were about equal.
The code and zoomable plots were made available so the same test can be repeated on other models.
Key points
- q8/q8 had almost no quality loss on both tested models.
- q4/q4 worked reasonably on Qwen but hurt Gemma badly.
- turbo4 was mixed and did not clearly beat q4_0.
- turbo3 and turbo2 the cache heavily but came with high quality loss.
- K and V sensitivity changed depending on the model and setting.