Reproduce “RATTENTION: Towards the Minimal Sliding Window Size in Local-Global Attention Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bailin Wang, Chang Lan, Chong Wang, Ruoming Pang
RATTENTION introduces a hybrid attention mechanism combining Sliding Window Attention (SWA) with a specialized Residual Linear Attention (RLA) to enable significantly smaller local window sizes in Transformer models. This approach allows local-global models to match or exceed full attention performance with minimal window sizes (e.g., 512 tokens), leading to substantial inference efficiency gains, including theoretical speedups of up to 60% for larger batch sizes.
Key insights
- Effectively combining a local attention mechanism with a recurrent linear attention mechanism designed to specifically capture 'out-of-window' context can overcome performance degradation from small local windows.
- The use of Residual Linear Attention (RLA) with its specific readout from an aggregated historical state is critical for maintaining performance while dramatically reducing the SWA window size.
- RATTENTION's architecture, by forcing generalization during pretraining with smaller local windows, implicitly improves zero-shot generalization to contexts significantly longer than the training context.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.