llama.cpp settings can free more GPU memory
Recent llama.cpp runs appear to use regular computer memory better, with no visible , making it easier to keep and longer context on the GPU. In one RTX 3090 external GPU setup, a Qwen 27B model could run with a 150k context while using settings that push most work onto the GPU and avoid normal RAM.
For models that handle images, moving the to the CPU can free about 1GB of GPU memory. The tradeoff is a small speed drop.
Changing the to a smaller format can cut memory use by half or more, but it can also reduce answer quality. After attention rotation was added, q4 cache compression seemed to keep quality acceptable, and the saved memory could be used for a larger base model, which may improve the overall result.
Key points
- Recent llama.cpp behavior appears to reduce normal RAM pressure and avoid visible .
- Moving the to the CPU can free about 1GB of GPU memory for s.
- Smaller formats can save a lot of memory but may lower answer quality.
- The saved GPU memory can be used for longer context or a larger model.
- For , these settings may help reduce by making existing hardware go further.