MRU tests a cheaper path than attention for long AI inputs
MRU is a sequence design meant to replace attention for data that comes in order, such as text or code. It turns each input embedding into a matrix-shaped state, multiplies those matrices across the sequence, creates an output state, and then turns that state back into a vector.
A was added so the method can run more efficiently on hardware. Earlier tests looked good on a simple Shakespeare character dataset, but harder datasets exposed two problems: the matrix states needed better limits, and training could become unstable.
New experiments change how the input state matrix is built. The original method reshaped the input vector into a matrix and added an identity matrix, while newer methods include building a skew-symmetric matrix from the vector and applying transforms such as the matrix exponential.
Key points
- MRU is proposed as a alternative to attention.
- It converts input into matrix states, multiplies them through the sequence, and converts the result back to a vector.
- A is used to make the method more suitable for hardware.
- Simple tests worked better than harder datasets, where training instability appeared.
- The update focuses on safer ways to create the input state matrix.