Speculative decoding needs better benchmarks than acceptance rate
has a formal guarantee: with , the final output stays the same as the larger target model. The quality of the smaller draft model does not decide correctness.
A weak draft model gets rejected more often, so it can reduce the speed benefit, but it does not change what the target model would say. This makes it different from , where speed or size gains can come with a quality tradeoff.
The is directly tied to the gap between the draft model and target model s: it equals 1 minus the . The practical question is why a lossless speed method is not the default in more local setups, and what real speed gains different draft-target pairs actually deliver.
Key points
- uses a smaller draft model to propose tokens before a larger target them.
- keeps the final output aligned with the target model.
- A poor draft model mainly hurts speed, not output quality.
- equals 1 minus between the two models' s.
- Benchmarks should report real speed and cost impact, not only .