A 6-GPU home setup runs a large local AI model
MiniMax M2.7_Q3_XL was run on a home server with six P40 s. The machine used an Asus X99-E-WS , a Xeon E5-2680 v4 2.40GHz CPU, 128GB of , an SSD, and six P40 cards.
The six cards provided 144GB of , and the BIOS was modified so it could handle many s. With a 32,768-token context, the basic setup handled input and generation at usable speeds, but performance dropped sharply at 65,536 tokens and 126,720 tokens.
Q8 KV was slower than F16 KV for use, with generation about 12.8% slower at 126,720 tokens. Reducing the batch size from 2048 to 1024 and the smaller batch from 512 to 256 improved input handling at the 32,768-token setting.
Key points
- Six P40 cards provided 144GB of for a large .
- Speed fell sharply as context grew from 32,768 tokens to 126,720 tokens.
- Q8 KV was slower than F16 KV for generation.
- At 126,720 tokens, Q8 KV generation was about 12.8% slower.
- Changing batch sizes improved input processing at the shorter context setting.