A faster long-context attention method could cut LLM compute
A firsthand experiment claims a new attention method can make much faster when they handle very , from 128,000 tokens to 1 million tokens. The model’s normal dense attention layer was removed and replaced with the new method. The comparison looked at attention FLOPs, real response delay, perplexity, and .
At 128,000 tokens, the attention compute was said to be about 42 times lower. Depending on the , the claimed speedup was about 6 times to 40 times. The tested prompts reportedly kept performance close to the unchanged model while using much less compute.
The claim still needs public code, outside testing, repeatable benchmarks, and proof on real .
Key points
- The method targets from 128,000 tokens up to 1 million tokens.
- Dense attention was replaced with a new attention method.
- At 128,000 tokens, attention compute was claimed to be about 42 times lower.
- The claimed speedup ranges from about 6 times to 40 times.
- Public code, outside testing, and real agent benchmarks are still missing.