Reproduce “Memory-efficient Transformers via Top-$k$ Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, Jonathan Berant
Top-k Attention, developed by researchers from IBM, Tel Aviv University, and AI2, reduces Transformer memory consumption to linear scaling with sequence length. This activation-agnostic method enables processing sequences up to 32K tokens on a 30GiB GPU, achieving an 8x memory reduction over vanilla Transformers, while maintaining comparable accuracy on diverse NLP tasks.
Key insights
- Selecting the exact top-k similarity scores for each query provides an effective sparse approximation that closely matches vanilla attention's accuracy.
- A custom memory-efficient backward pass, leveraging input checkpointing of `top_dots` and `top_indices`, is crucial for achieving practical linear memory scaling during training.
- Feed-forward layers can be reframed as attention mechanisms, enabling Top-k Attention to significantly reduce their memory footprint, particularly in wide models, and acts as a drop-in replacement for pre-trained models without corrective pre-training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.