Reproduce “Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kewei Li, Yanwen Kong, Yiping Xu, Jianlin Su, Lan Huang, Ruochi Zhang, Fengfeng Zhou
This research introduces InfoScale and CosScale, theoretically derived scaling factors based on information entropy invariance, to enhance length extrapolation capabilities in attention mechanisms. The proposed methods achieve state-of-the-art performance on the GAU-α model, extending context windows up to 64 times the training length, and provide insights into the role of attention score dilution in limiting long-range context handling.
Key insights
- Maintaining information entropy invariance in attention scores is crucial for effective length extrapolation, providing a theoretical foundation for scaling factors.
- Enforcing that word embeddings lie on a hypersphere (as with cosine attention) enhances the effectiveness of entropy-preserving scaling methods and improves angular training of the attention matrix.
- Attention score dilution, caused by new tokens in long contexts, is identified as a primary bottleneck limiting the performance of attention mechanisms in long-range modeling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.