Using an RTX 5080 with an RTX 4060 for faster local AI inference
An RTX 4060 with 8GB of memory is currently being used for with Qwen 3.6-35B-A3B Q8. The setup keeps weights, values, and keys at Q8 for better quality, allows up to 60k context per agent, and uses CPU plus DDR4 memory offloading because the memory is limited. Even when the context is still low, speed is only about 100pp for and 20tg for text generation.
Lowering weights to Q4 raised generation speed to about 30tg, but the quality loss made Q8 preferable. The next target is running Qwen 27B at least at Q4 to Q6 quality, with 20tg minimum and ideally 30 to 40tg or more. The same computer is also used for demanding PCVR games, so one large is preferred over several smaller cards.
The motherboard has a PCIe 5 x16 slot and a PCIe 3 x16 slot, and the possible upgrade is to move the RTX 4060 to the slower slot and put a new RTX 5080 in the main slot. The budget for the and a new is 1,500 to 2,000 euros, with the lower half of that range being more comfortable.
Key points
- An RTX 4060 8GB is being used for with Qwen 3.6-35B-A3B Q8.
- The setup uses up to 60k context per agent and relies on CPU and DDR4 memory offloading.
- Current speed is about 100pp for and 20tg for text generation when context is still low.
- Q4 weights improved generation speed to about 30tg, but quality was worse.
- The planned upgrade is an RTX 5080 plus the existing RTX 4060, with a total budget of 1,500 to 2,000 euros including a new .