Testing a tiny autocomplete model on 32GB of VRAM
A 32GB VRAM setup is too limited for building a large general-purpose AI model, so the goal is narrowed to a small model. The planned size is about 25 million .
The model would avoid full chat-style answers and instead predict the next token, sentence, or paragraph from a given context. The main challenge is .
Using a rough rule that training needs several times more tokens than the parameter count, even a 25 million parameter model may need more than 100 million tokens for experiments. One first test idea is to clean YouTube comedy and train the model to continue setup-to-punchline patterns.
Key points
- 32GB of VRAM makes a large general model unrealistic, but a small focused model may be possible.
- The target is around 25 million .
- The model would focus on instead of full chat responses.
- Even a small model may need more than 100 million tokens for useful experiments.
- Clean, narrow data such as comedy could make the first test easier.