A small tool trims an LLM KV cache when memory gets tight
kv-psi is a small reference that uses to shrink an LLM KV cache when a computer is under . A KV cache stores past context so the model can keep generating text without recalculating everything, and it grows during long chats or long document work. The tool reads from cgroup memory.pressure or /proc/pressure/memory, then prunes part of the cache when pressure rises.
It needs Linux with PSI enabled, Python 3.10 or newer, llama.cpp runner build , and a GGUF model. The basic flow is to run a simulator, build the llama.cpp runner, and download a small benchmark model if needed. In a recent Jetson result with 1536 decoded tokens, the fixed setup did no pruning and ended with a final KV size of 1547, while the PSI setup pruned 2 to 4 times and ended at 1004 or 1291.
Speed varied by run order and pressure state: PSI showed 88.80 or 96.16 , while the fixed setup showed 94.00 or 89.76 . The benchmark runs both variant orders because cache state, zram/swap, and PSI averages can carry over from one pressure run to the next.
Key points
- is used as the signal for when is high.
- The tool prunes the KV cache instead of keeping all past context in memory.
- In the Jetson result, PSI pruning reduced the final KV size to 1004 or 1291, compared with 1547 for the fixed setup.
- Token speed was mixed, so the result is more about memory control than guaranteed speed gains.
- Requirements include PSI-enabled Linux, Python 3.10+, llama.cpp, and a GGUF model.