A small way to shrink Q4_0 model files by reusing scale values
Q4_0 files can contain many repeated scale values. This repetition was found in Qwen 3.5 2B and . The idea is to store a number that points to a shared scale list instead of storing the full scale value every time.
For the roughly 15GB Q4 model, the estimated minimum saving is 318MB. The example only looks at part of the model: ffn_down, ffn_gate, and ffn_up, each with 89,128,960 weights across a 64-layer model. In Q4_0, every 32 weights has one 16-bit BF16 scale, which creates 2,785,280 scale entries for one such sub-layer.
Other sub-layers may offer more room to save space. The tradeoff is that needs custom code to turn the stored index back into the right scale value.
Key points
- Repeated scale values were found in Qwen 3.5 2B and .
- The proposed method stores an index to a shared scale list instead of repeating each scale value.
- The estimated minimum saving for Q4 is 318MB.
- In Q4_0, every 32 weights gets one BF16 scale value.
- Using this in practice would require custom code.