Keeping model weights compressed can make CPU LLMs much faster
A CPU-only C99 engine was much slower than llama.cpp on Q4_K_S. On the same i5-11300H laptop with 4 threads, the custom engine processed 1.90 , while llama.cpp processed 13.79 . That is a 7.3 times gap.
The likely cause was not raw calculation power, but how much data had to be read from memory. The custom engine expanded Q4K weights into F32 before running some model parts, so each weight used 4 bytes during inference, and it also expanded MoE expert weights during each call. llama.cpp kept reading the compact Q4K data directly, using about 0.5 bytes per weight, and combined that with a Q8-compressed activation vector in one pass.
This avoided a large F32 middle step. Changing SIMD options such as AVX2, AVX512F, and VNNI only changed speed by about 2%, because the main limit was memory reading, not math instructions.
Key points
- On the same CPU and 4 threads, the custom engine reached 1.90 , while llama.cpp reached 13.79.
- The main speed gap came from reading much more data from memory.
- Expanding Q4K weights into F32 changes about 0.5 bytes per weight into 4 bytes per weight.
- llama.cpp calculates directly from compact Q4K weights and Q8 without a large F32 middle step.
- Changing SIMD options did little because memory reading, not math speed, was the .