A tiny open vision-language model for learning how image-to-text works

SupraLabs released SupraVL-Nano-900k, its first . It has about 900,000 and was trained from scratch on the Flickr8k dataset. It is not meant to be a production model.

It is meant to be a clear, readable blueprint for understanding how image-to-text models work inside. The whole fits in one . The image side uses a CNN visual encoder, and the text side uses a GPT-2-style decoder.

Its tokenizer was trained on the Flickr8k captions and uses 2,048 BPE tokens. The model combines 16 s and 48 text tokens, for 64 total positions.

Key points

  • SupraVL-Nano-900k is a small with about 900,000 .
  • It was trained from scratch on Flickr8k and fits in one .
  • The image part uses a CNN visual encoder, while the text part uses a GPT-2-style decoder.
  • It uses 16 s and 48 text tokens, for 64 total positions.
  • The release is mainly for learning, not for production use.
Read original