A small GPT-style model built directly in C and CUDA
nanoeuler is a GPT-2-style built in C and CUDA without PyTorch or other libraries. The repository directly implements the forward pass, , , pretraining, and SFT. The small CPU model has about 1.05 million parameters, while the GPU pipeline trains a roughly 116 million parameter model on one RTX 4070.
The GPU version includes cuBLAS, a hand-written , RMSNorm, RoPE, grouped-query attention, SwiGLU, and AdamW. is presented as making each training step about 3 times faster. The resulting chat model is not a useful assistant: it can produce assistant-shaped replies, but its knowledge is shallow and often wrong.
The project is mainly an educational, from-scratch view of how a training pipeline works end to end.
Key points
- The model is built from scratch in C and CUDA, without PyTorch.
- The GPU model has about 116 million parameters and trains on one RTX 4070.
- The hand-written is reported to make training steps about 3 times faster.
- SFT turns the pretrained model into a chat-shaped model, but the answers remain shallow.
- This is most useful for understanding model cost and speed mechanics, not for deploying an agent.