Open handbook explains how to run LLM inference at scale
An open handbook is being built to explain how run in real services. The newest chapter covers GPU execution and memory internals.
It focuses on why a GPU can sit partly unused during inference, how the memory hierarchy limits , and where the real slowdowns happen. The handbook also covers KV cache, batching, vLLM, SGLang, and TensorRT-LLM, which are all tied to faster and cheaper model serving.
Diagrams made with mermaid are used to make the system flow easier to follow. The handbook is still in progress, and feedback is being requested from people who have run inference in production.
Key points
- The handbook explains at scale in an open, work-in-progress format.
- The latest chapter covers GPU execution and memory internals.
- It explains why GPUs may be underused during inference and where appear.
- It includes KV cache, batching, vLLM, SGLang, and TensorRT-LLM.
- Production feedback and corrections are being invited.