Claude helped build a small looped language model from scratch
Claude was used as a coding and research helper to build a 135 million parameter from the ground up. The model was trained on 4.6 billion tokens from FineWeb, with experiments inspired by the Parcae paper on making s more stable.
Five comparison tests took about two weeks to debug, but at this small size, the more complex stability methods did not beat the simple baseline. The training was still completed, followed by , and both the base model and tuned model were released on .
Training used two H100 s on Modal plus free Lightning H200 time, took about three hours, and cost about $51 in total. Claude helped check the paper , find optimizer routing mistakes, write the Modal training setup, and get the model shipped.
Key points
- A 135 million parameter was trained from scratch.
- The model used 4.6 billion FineWeb tokens, then received .
- Five Parcae-style stability tests did not outperform the simple baseline at this size.
- Training took about three hours on two H100 s and cost about $51.
- Claude helped with paper , optimizer bugs, Modal training code, and release work.