How far a 16GB-class GPU can go for local AI models
A local AI setup with an RTX 5080 and 32GB of is being used to test what model size is realistic. at Q3KM reaches about 60 , while at IQ2 reaches about 190 . Moving down from Q6 to smaller formats can cause a noticeable loss in accuracy.
A spare 1080 Ti with 11GB of GDDR5X memory could be added so the model is split across two s. The main question is whether that extra 11GB of is worth the extra power use, heat, and setup complexity.
Key points
- The setup uses an RTX 5080 with 32GB of .
- at Q3KM runs at about 60 .
- at IQ2 runs at about 190 .
- Dropping below Q6 may reduce accuracy in a noticeable way.
- Adding a spare 1080 Ti could provide 11GB more GDDR5X memory, but with extra power and setup costs.