Reproduce “Lite Transformer with Long-Short Range Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, Song Han
Researchers from MIT developed Lite Transformer, an efficient architecture featuring Long-Short Range Attention (LSRA) tailored for natural language processing on resource-constrained mobile devices. It demonstrated improved performance over standard Transformers under mobile constraints and achieved competitive results against AutoML-designed models with significantly lower computational and environmental costs.
Key insights
- For 1-D sequence NLP tasks, the Feed-Forward Network (FFN) layers, not attention, represent the primary computational bottleneck in standard Transformer blocks.
- Specializing attention into distinct components for global (standard attention) and local (convolutional) context capture, using channel-wise splitting, enhances overall efficiency and contextual modeling capacity.
- Carefully human-designed architectures, informed by systematic analysis, can outperform computationally expensive AutoML-based designs (e.g., Evolved Transformer) with a drastically lower carbon footprint and development cost.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.