A practical question about running big local models at 4-bit
The hardware under consideration is a setup with 4 to 8 AMD 6000 PRO s. That would provide about 384GB to 768GB of . The goal is to run large models such as GLM 5.2, Kimi 2.7, and locally.
In theory, these models may fit at 4-bit, while 8-bit may need too much memory. The main concern is whether 4-bit causes a large quality drop for agent work or programming compared with 8-bit. Existing benchmark data is available, but it does not cover some of the newest models.
vLLM, SGLang, and other backends are possible ways to run the models.
Key points
- A 4-card to 8-card 6000 PRO setup would offer about 384GB to 768GB of .
- The include GLM 5.2, Kimi 2.7, and .
- 4-bit may make very large models fit into memory, but quality may drop versus 8-bit.
- Agent work and programming are the main being questioned.
- vLLM, SGLang, or another backend could affect real-world speed and cost.