Running a 35B AI model on a 16GB Mac by streaming from SSD
A developer extended the open-source ds4 runtime to run the model on a 16GB M1 Pro Mac. The model's Q4_K_S quantized version is about 20.8GB, too large to fully fit in 16GB of memory. The solution, called DS4, keeps only a limited cache of frequently used "expert" modules in memory and streams the rest from SSD as needed — this works because the model uses a , where only a subset of expert modules is used per request.
On the 16GB M1 Pro, a four-run warm median measured 15.04 for prefill (initial input processing) and 9.77 for text generation. stayed normal throughout, and no swapping (offloading memory to disk) occurred. The repository includes the runtime code, the GGUF file format converter, benchmark details, and links to the published GGUF model files.
The approach is still , but it now works on an actual 16GB machine rather than requiring a larger Mac.
Key points
- Runs (Q4_K_S quantized, ~20.8GB) on a 16GB Mac
- DS4 approach: caches frequently used expert modules in memory, streams the rest from SSD
- On 16GB M1 Pro: 15.04 tokens/s prefill, 9.77 tokens/s generation (4-run warm median)
- stayed normal, no swapping observed during testing
- Runtime code, GGUF converter, benchmarks, and model file links published on GitHub