Open softmax-free language model targets long-context memory savings
RRT-355M is a shaped like GPT-2 Medium, with about 354 million parameters and training on 11.5 billion tokens from scratch. The main experiment is whether an attention system can work without softmax, while turning off many weak token-to-token links. The model comes with and custom Triton kernels that can skip chunks of work during inference.
On the 22-task CORE benchmark, it scored 0.1558, below GPT-2 Medium at 0.1770 but above GPT-2 124M at 0.1211. In tests, the kernel skipped 34% to 55% of tiles on an H100, and a 16,384-token attention run was reported at 5.5GB peak VRAM. It cannot be run correctly as a normal s GPT-2 model; it needs the separate RRT engine and .
MMLU, GSM8K, HumanEval, chat behavior, instruction following, and fine-tuned tasks were not evaluated, and the code is under AGPL-3.0.
Key points
- The model has about 354 million parameters and was trained on 11.5 billion tokens.
- It tests attention without softmax and turns off many weak token links.
- Custom Triton kernels skip work in inference to reduce VRAM use.
- CORE score is 0.1558, below GPT-2 Medium but above GPT-2 124M.
- It needs a separate RRT engine and cannot be used as a normal s GPT-2 model.