Why long context can slow large language models
can slow down on long context because memory movement, not raw computing power, becomes the main limit. During , the model must keep reading the Key-Value cache from earlier tokens while it produces the next token.
That data has to move from into much faster memory near the processor, and this transfer can make the chip wait. As context grows to thousands of tokens, standard Multi-Head Attention creates heavy memory traffic because each Query head has its own Key/Value partner.
Multi-Query Attention reduces that load by letting all Query heads share one Key/Value pair. Newer model designs such as LLaMA 3 and Mistral follow this broader push to get around the memory bottleneck.
Key points
- Long context can make limited by rather than compute power.
- repeatedly reads the Key-Value cache from earlier tokens.
- Standard Multi-Head Attention increases memory traffic as context gets longer.
- Multi-Query Attention cuts memory load by sharing one Key/Value pair across Query heads.
- Agent builders can reduce cost by trimming context and choosing efficient model designs.