ExLlamaV3 hits 1.0, bringing major speed and compatibility upgrades
ExLlamaV3, an for running locally, has reached its first , version 1.0.0, after more than a year of development. The update removes external dependencies on flash-attention-2 and xformers, simplifying installation and builds.
support — splitting a model's computation across multiple GPUs — now covers most model types, including Gemma4. A new attention kernel enables online cache quantization without the usual slowdown, and in some cases actually speeds up inference.
GEMM/GEMV matrix-multiplication performance, core to how these models run, has been significantly improved on Nvidia Ampere-generation GPUs. The release also adds a new INT8 GEMV kernel, an improved scheduler for models, and support for new s including GptOssForCausalLM and NemotronHForCausalLM, alongside numerous smaller optimizations, bug fixes, and faster build times.
Key points
- ExLlamaV3 reaches its first , 1.0.0, after over a year of development
- Removed flash-attention-2 and xformers dependencies, simplifying installation
- KV cache quantization no longer slows inference and can even speed it up
- support extended to most models, including Gemma4
- Adds support for new architectures like GptOssForCausalLM and NemotronHForCausalLM