H64LM shows a small open LLM built to study cost-saving designs
H64LM is a 249-million-parameter model built directly in PyTorch. Its goal is to help study how modern work, not to provide a strong ready-to-use model. The core parts were written manually instead of using a high-level training framework, including attention, expert routing, normalization, and the training loop.
The design includes grouped query attention, a sparse setup with 8 experts and top-2 routing, three helper losses for routing, SwiGLU, RoPE, RMSNorm, and attention. The training setup includes mixed-precision training, gradient accumulation, a custom training loop, checkpoint saving, and resume support. The released checkpoint was trained on part of WikiText-103 only to prove the pipeline works end to end; it is visibly overfit after epoch 10, with a best of about 40.5.
Current limits include generation with batch size 1 only and no true DDP, with DataParallel used instead.
Key points
- H64LM is a 249-million-parameter model built from scratch in PyTorch.
- It uses a sparse design with 8 experts and top-2 routing.
- It includes efficiency-related ideas such as grouped query attention, attention, and mixed-precision training.
- The included checkpoint is for pipeline testing, not strong model performance; best is about 40.5.
- Generation currently works only with batch size 1, and true DDP is not supported.