Predicting MoE expert routing to speed CPU/GPU offload 5x
Running (MXFP4 GGUF) on an RTX 3060 12GB with CPU/GPU offloading — where some expert layers sit in RAM and get pulled over PCIe as needed — the GPU typically sits idle waiting for experts to arrive, since PCIe bandwidth, not compute, is the real bottleneck. The proposed fix uses the model's own head, normally used for to draft the next token early.
Instead of only using that draft for accept/reject decisions, the approach also checks which experts the drafted token would route to, then prefetches those experts in the background on a separate CUDA stream. This hides PCIe latency behind ongoing compute rather than paying that latency on every single token.
Before building anything, the author instrumented llama.cpp to check whether the idea was worth pursuing at all, using Fable 5 and GPT-5.6 to help analyze the numbers. The result: a projected speedup from 30 tokens/sec to 150-200 tokens/sec.
Key points
- Tested on (MXFP4 GGUF) with CPU/GPU offload on an RTX 3060 12GB
- Current bottleneck is PCIe bandwidth, not GPU compute — GPU idles waiting for expert data
- Uses the MTP head to predict next-token expert routing early, prefetching those experts on a separate CUDA stream
- Verified feasibility via actual llama.cpp before implementing
- Claims a theoretical speedup from 30 tokens/sec to 150-200 tokens/sec