Reproduce “RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiuying Wei, Anunay Yadav, Razvan Pascanu, Caglar Gulcehre
The Recurrent Attention Transformer (RAT) architecture combines intra-chunk recurrence with inter-chunk softmax attention, enabling efficient processing of long sequences. It achieves up to 9x faster inference and 15x higher throughput compared to full attention, while maintaining or improving accuracy on various long-context language tasks and reducing KV cache memory usage.
Key insights
- Combining intra-chunk recurrence for local information compression with inter-chunk softmax attention for global access effectively balances computational efficiency and full attention's representational power.
- A tunable chunk size allows for flexible interpolation between purely recurrent and full attention behaviors, offering architectural adaptability.
- The design, including chunk-level RoPE and an online softmax formulation for causal masking, facilitates efficient parallel training and inference without custom CUDA kernels.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.