A 4-card home LLM setup aimed at cutting token costs
A personal local AI server is built with four 16GB s. The hardware is stacked on a kitchen-style shelf instead of a standard , with the main slot split and a riser cable used to connect the cards. The setup runs two llama.cpp instances at the same time and uses a Qwen model in a smaller, lower-memory format.
Each instance is set up for about 150,000 tokens of context, with input processing around 1,000 and answer generation around 45 to 60 . The machine uses an i5 processor and 32GB of , but the model work mainly runs in memory. opencode was used to build a backend that manages llama.cpp and counts tokens.
A rough cost check suggests about $60 has already been saved. The system is still buggy, and the next step is a router that sends different parallel jobs to different sets of cards and servers.
Key points
- The setup uses four 16GB s for running.
- It runs two llama.cpp instances with a Qwen model.
- Reported speed is about 1,000 for input and 45 to 60 for output.
- opencode was used to build a backend for model management and token counting.
- The rough claimed saving is about $60 so far.