JetSpec reports up to 9.64x faster LLM response generation
JetSpec is a research system for making produce answers faster. It prepares many possible next-token paths in a tree-like shape in one pass, then lets the main model accept only the paths that match its normal output. The reported speedup is up to 9.64x on MATH-500 and 4.58x for open-ended chat, without changing the final answers.
With CUDA graph and , the system reaches about 1000 on one B200 GPU. Earlier methods often faced a tradeoff: better draft choices became expensive as the tree got deeper, while cheaper draft choices could become inconsistent across branches. JetSpec tries to avoid that by building a causality-preserving tree in a single pass.
The project page, code, and blog are public, so others can inspect and test the method.
Key points
- JetSpec focuses on faster inference for .
- It reports up to 9.64x speedup on MATH-500 and 4.58x on open-ended chat.
- It reports about 1000 on one B200 GPU after low-level .
- The method claims to keep the model’s final answers unchanged.
- The project page, code, and blog are available for review.