What actually happens between hitting enter and an LLM's first word

After you type a prompt and hit enter, an LLM generates its response one token (a word or word-fragment) at a time. Because each new token requires recomputing over all previous tokens, this process is inherently slow. A technique called the KV cache addresses this by storing previously computed results so they can be reused instead of recalculated from scratch for every new token.

Decoding strategies like temperature, top-k, and top-p then control how predictable or varied the model's word choices are at each step. This is the final part of a four-part series explaining how LLMs work: earlier parts covered tokenization, embeddings, and the forward pass; the loss function, backpropagation, and optimizers; and scaling, fine-tuning, RLHF, and DPO.