FP8 MoE models fail in vLLM on an AMD Hermes-Agent server
vLLM replaced Ollama on a server because Ollama did not work smoothly with . The machine is a GekTec EVO-X2 with an AMD Ryzen AI Max+ 395, Radeon 8060S, and 128GB of . Its BIOS assigns 32GB to the system and 96GB as .
After substantial Docker setup work, it can run the 26-billion- Gemma 4 model using about 70GB of . However, vLLM stops before downloading the model files for an FP8 version of the 35-billion- Qwen 3.6 MoE model. The log reports that no backend is implemented for FP8 MoE.
It remains unconfirmed whether AMD ROCm builds of vLLM can run this kind of model inside Docker or which would make it work.
Key points
- vLLM was used because Ollama did not work smoothly with .
- The server assigns 96GB of its 128GB as .
- Gemma 4 26B runs while using about 70GB of .
- The FP8 MoE model fails before its files are downloaded.
- No proven Docker or solution is provided.