RWKV: an RNN that matches Transformer performance and trains in parallel
RWKV is a recurrent (RNN) architecture that achieves performance on par with Transformers. Standard Transformers require compute and memory that grow quadratically as input length increases, but RWKV, being RNN-based, scales only linearly with in theory.
At the same time, RWKV is designed to support like a Transformer, overcoming the slow sequential training that normally limits RNNs. A user posted introducing this architecture.
Key points
- RWKV is RNN-based but reaches Transformer-level LLM performance
- Transformers scale quadratically with input length; RWKV does not
- RWKV supports , unlike traditional RNNs