Flash-MSA opens faster training for million-token AI models
Flash-MSA is an unofficial open-source kernel for efficiently training MiniMax . It runs on H100 and B200 GPUs and uses a small selector to choose relevant information in blocks of 128 instead of repeatedly processing the entire context. It saves the selected block locations for reuse, allowing every part of a training step except the selector’s first pass to scale linearly with .
It also calculates the same training gradient without storing the full KL divergence, reducing memory traffic and extra work. In bf16 correctness tests at 4,096 and 8,192 tokens, its outputs and gradients had scores of about 0.9983 to 1.0000 against a standard PyTorch version. Installable code and a Megatron-LM training example are available, but this is not an official MiniMax .
Current limits include fixed head and block sizes of 128, no quantized training, and missing support for some attention features. Training at one million tokens also requires unfinished work for splitting the context across several GPUs.
Key points
- The selector sends only relevant 128-token blocks to the main attention calculation.
- Selected block locations are cached and reused during .
- bf16 tests reached of 0.9983 to 1.0000 against a PyTorch reference.
- Open-source H100 and B200 code includes a Megatron-LM training example.
- Quantized training and complete multi-GPU support for million-token training are not yet available.