HDD-RoPE tests a faster-learning position method
HDD-RoPE is a new way to help a understand word order. Standard RoPE splits into pairs and rotates each pair at a set rate, so the model can learn relative position from how those values change.
HDD-RoPE takes the idea from an earlier project and uses it as a position method. In a TinyStories training test, a GPT-2-like model using HDD-RoPE reached lower faster than a baseline using xPos.
The test model followed TinyStories-33M-style settings, with 4 blocks and a 768-size model dimension. The public repository includes code to reproduce the result, plus math and details.
Key points
- HDD-RoPE is a position method for .
- Standard RoPE rotates paired query and key values to represent relative position.
- HDD-RoPE uses a as the basis for position handling.
- On TinyStories, it lowered faster than an xPos baseline .
- The code and math notes are available in a public repository for replication.