Picotron helps train language models on older GPUs
Picotron is an open-source training framework meant to make training run on older or cheaper GPUs. Nanotron can fail before training starts because it loads heavy GPU-specific tools such as flash-attn, triton, and functorch as required parts of the code. Picotron removes those required hardware-specific with a fresh .
It should run on most GPUs that support PyTorch, using FP16 by default on older cards and BF16 on newer ones. If is installed, Picotron can use it at runtime; if not, it falls back to standard PyTorch attention. A tiny 2 million parameter model has been trained locally on FineWeb-Edu with it.
The included settings cover GQA, MLA, QK-Norm, logit soft-capping, parallel FFN/Attn runs, and ZeRO-1 wrapping on DDP. The short roadmap includes MoE preparation and easier dataset preparation.
Key points
- Picotron removes mandatory GPU-specific that can break on older cards.
- It targets older or budget GPUs such as T4 and V100.
- It works with standard PyTorch attention by default and uses only when available.
- A tiny 2 million parameter model was trained locally on FineWeb-Edu.
- It may lower experiment costs for agent-related model work, but it is not a direct token-saving tool.