A 6GB laptop GPU ran a large local AI model
ran on a 2019 gaming laptop at about 28 . The machine used an i7-9750H processor, a GTX 1660 Ti with 6GB of video memory, and 32GB of system memory. The model has 35 billion total parameters, but its MoE design uses only about 3 billion for each token, so each step is much lighter than the full model size suggests.
The `--n-cpu-moe 36` setting kept attention and shared tensors on the GPU while moving expert tensors into system memory. Turboquant’s `turbo4` and `turbo3` KV cache quantization made the long 128K context fit into only 6GB of video memory. The model was exposed on `localhost:8080` as an and connected to opencode.
That means local agent-style coding work could run without cloud use or API bills. The Turboquant fork changes quickly, so the specific commit `4595fff` was recommended for .
Key points
- ran at about 28 on a GTX 1660 Ti with 6GB of video memory.
- The MoE design means only about 3 billion parameters are active for each token, even though the full model is 35 billion parameters.
- `--n-cpu-moe 36` reduced GPU by moving expert tensors into system memory.
- `turbo4` and `turbo3` KV cache quantization helped fit a 128K into 6GB of video memory.
- An on `localhost:8080` made it usable with opencode without cloud API bills.