llama.cpp patch brings multi-token prediction to older GPUs
in could select BF16 calculations without properly checking whether an older GPU supported them, causing cuBLAS to crash on hardware such as Kepler cards. A CUDA backend patch now uses BF16 when available, falls back to fast FP16 on suitable cards, and uses FP32 on still older hardware. This keeps the feature working across Kepler, Maxwell, Pascal, and Turing without changing behavior on newer GPUs.
A test used a model with Q4XL and four overclocked Tesla K40c cards. Speed was about 22 with stock , 17.5 with disabled, and 25 with the patched feature predicting two tokens at a time.
Key points
- The failure came from selecting BF16 on GPUs that do not support it.
- The patch chooses BF16, FP16, or FP32 according to the GPU's abilities.
- Four Tesla K40c cards reached about 25 with two-token prediction, compared with 17.5 when it was disabled.
- The change is designed to leave newer GPU behavior unchanged.
- Reusing older hardware could improve AI agent speed and lower the cost per task.