Running 100B-plus AI models on an old, low-spec laptop
A laptop with a Core i7-8750H, 20GB of memory, and a GTX 1050 Mobile with 4GB of was able to run models with more than 100 billion parameters. The setup avoided , which use every part of the model, and used MoE models that activate only selected expert parts for each task. Most model data stayed on a 512GB Samsung NVMe drive, while mmap loaded the needed portions and the processor handled the experts.
The KV cache used Q4_0 , and downloaded model files usually used Q3. For models above 700 billion parameters with more than 20 billion , the setup did not use anything below Q2. In LM Studio, UD-IQ3_XXS produced about 1.0 to 1.8 , while Nemotron-3-Super-120B-A12B UD-Q3_K_M reached about 1.5 to 2 .
Responses were very slow, but batch-style jobs completed useful , code repair, and code review work while the machine was left unattended.
Key points
- Use MoE models so only selected expert parts run for each task.
- Keep most model data on an NVMe drive and use mmap to load required portions.
- Use Q4_0 for the KV cache and usually Q3 for the model files.
- Expect roughly one to two generated on this hardware.
- Group work into unattended batches instead of waiting for an interactive reply.