A mixed-GPU setup question for running hermes agent locally
This is a real hardware setup for running A3B Q8_K_XL with . The goal is to use a model believed to fit well in a 96-128GB VRAM setup. The available hardware is split across two machines.
One machine has two NVIDIA RTX 3090 cards, an ASRock Rack ROMED8-2T , an EPYC 7262 CPU, and 64GB of . The other has a 9900X CPU, 32GB of , a Radeon 7900 XTX, and a Radeon 7900 XT. The main practical idea is that mixed NVIDIA and AMD cards may need with the Vulkan backend to work together.
There is no confirmed success result yet; this is a setup being evaluated.
Key points
- The is A3B Q8_K_XL.
- The setup is aimed at with around 96-128GB VRAM.
- One machine has dual RTX 3090 cards, EPYC 7262, and 64GB .
- The second machine has Radeon 7900 XTX and 7900 XT cards with 32GB .
- with the Vulkan backend may be needed for mixed NVIDIA and AMD use.