Speculative decoding raised energy use in one GPU test
A firsthand test measured by actual power use, not just speed, and the result was worse than expected. It did not make faster, and it used more energy for each token. The setup used an RX 6650 XT with 8 GB of memory, Windows 11, llama.cpp with Vulkan, and with 8 slots plus continuous batching.
The main model was Qwen2.5-3B-Instruct in a quantized version, and the smaller was Qwen2.5-0.5B-Instruct in a quantized version. Both models ran fully on the GPU. The same 8 prompts were tested each time, each request generated 256 tokens, was turned off, and both 1-stream and 8-stream cases were measured.
GPU power was sampled with LibreHardwareMonitor about 6 times per second, then converted into energy used during each generation window. was about 19 watts, and the energy penalty became worse as GPU load increased.
Key points
- did not improve token speed in this test.
- Energy used per token was higher with enabled.
- The result got worse under heavier GPU load and more concurrent streams.
- The measurement used actual generation-time energy, not only average wattage.
- Cost-focused AI agent builders should test their own hardware before enabling it.