NextLat may make transformer inference up to 3.3x faster
Microsoft Research’s NextLat is a training method that tries to reduce the limits of predicting only the next token. A normal learns to guess the next token, while NextLat also trains it to predict its own next from the current and the next token.
This pushes the model to compress earlier information into a smaller belief state that can support reasoning and planning. Predicting in may also give the model richer training signals than only choosing one token from a large list.
The method can use recursive multi-step lookahead for , which is claimed to make inference up to 3.3 times faster. The item includes links to a blog, code, and paper.
Key points
- NextLat trains a to predict both the next token and its next .
- The goal is to help the model compress history into a compact belief state for reasoning and planning.
- prediction may provide denser training signals than token-only prediction.
- is claimed to make inference up to 3.3 times faster.
- For AI agents, faster inference could reduce repeated-call cost if quality stays stable.