Qwen 27B runs twice as fast with much lower memory use
Qwen3.6-27B Q4_K_M was tested on a single RTX 3090 with a 256K token context and generated 38.6 . The KV cache kept only 72MiB resident in memory, and total VRAM use fell from about 21GB to 17.5GB. On the same hardware, generation speed was reported to roughly double.
In a retrieval test, it found the target information with 88% to 100% recall while keeping only 6% resident. HumanEval, GSM, MATH, and agent test suites all stayed at 36/36 correct, matching the full KV cache result. Long outputs may not be exactly byte-for-byte identical because the optimized path rounds numbers differently, but the measured correctness stayed the same.
The code is available on GitHub in Lucebox Hub.
Key points
- Qwen3.6-27B Q4_K_M handled a 256K token context on one RTX 3090.
- Generation reached 38.6 and was reported to be about twice as fast on the same hardware.
- VRAM use dropped from about 21GB to 17.5GB.
- The KV cache used only 72MiB of resident memory, with 88% to 100% recall in a retrieval test.
- HumanEval, GSM, MATH, and agent suites matched the full KV cache result at 36/36 correct.
Sources covering this story (5)
- r/LocalLLaMAQwen 27B runs twice as fast with much lower memory use ↗
- r/LocalLLaMABest Settings for 48GB VRAM + Qwen 3.6 27B ↗
- r/LocalLLaMA7900XTX 24GB vram, can finally fit Q6K+MTP with Qwen 3.6 27B at 131k context ↗
- r/LocalLLaMA$1800 (in GPU cost running with P2P running Qwen/Qwen3.6-27b-FP8 with 262K context and BF16 KV cache at 55 tok/s ↗
- r/LocalLLaMACheapest hardware for Qwen 3.6: both 27B and 35B-A3B ↗