llama.cpp Vulkan setup runs a long-context model on a 7900 XTX
This setup runs IQ4_XS with a Vulkan build of llama.cpp on Ubuntu and an AMD Radeon RX 7900 XTX. It targets a very long 262k context and uses about 22GiB of memory.
Compared with an optimized ROCm 7.14 setup on the same hardware, and are said to be about twice as fast, with lower memory use. The shell script makes the llama-server path, model path, Vulkan device, host, port, context size, cache type, cache memory, GPU layers, thread count, and batch size configurable through .
The defaults include local access on 127.0.0.1:8081, context size 262144, q4_0 cache, 16384MB cache RAM, 99 GPU layers, 20 threads, and batch size 512. It requires a Vulkan-enabled llama.cpp build, a working Vulkan driver, and a GGUF model file.
Key points
- The setup uses llama.cpp with Vulkan on Ubuntu and an AMD Radeon RX 7900 XTX.
- It runs IQ4_XS at about 22GiB of memory use.
- The context is set to 262k, which is useful for longer .
- It is reported to be about twice as fast as an optimized ROCm 7.14 setup for and .
- The script lets users adjust server path, model path, port, cache, threads, and batch size through .