Gemma 12B runs much slower than expected in one local setup
Switching to Q8 cut generation speed from about 70 to about 10 . This happened even though the earlier model, GPT-OSS 20B Q4, was larger.
Trying a Q5 version of Gemma did not meaningfully improve speed compared with Q8. Turning off only added about 2 more .
The setup runs llama.cpp with 16 threads, an 8192 , 99 GPU layers, and a 4096 batch size. The GPU uses about 10GB of its available 20GB memory, so the slowdown does not look like a simple problem.
Key points
- Q8 dropped from about 70 to 10 in this setup.
- The Q5 version did not give a clear speed improvement over Q8.
- Disabling only added about 2 .
- The GPU used about 10GB out of 20GB, so memory capacity was not fully used.
- Agent builders should benchmark real before switching models.