What speculative decoding is, explained simply
is a technique that helps (LLMs) generate answers faster. Instead of relying only on the big, slow 'main model,' a small, fast 'draft model' guesses several words ahead first. The main model then checks all those guesses at once: correct guesses are accepted immediately, and as soon as a wrong guess appears, the main model takes over and generates from that point itself.
The key benefit is that the main model only needs to verify multiple words in one pass instead of producing them one at a time, which speeds up the overall process. The output quality stays the same as if the main model had generated everything on its own from the start, while response latency and cost both go down.
Key points
- A small draft model guesses several words ahead of time
- The large main model verifies all the guessed words in one pass
- Correct guesses are kept; the main model regenerates from the first wrong guess
- This is faster than generating one word at a time
- Output quality matches what the main model alone would have produced