Reproduce “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation” on OpenResearch

About this paper

Ofir Press, Noah A. Smith, Mike Lewis

Attention with Linear Biases (ALiBi) enables transformer models to effectively extrapolate to significantly longer input sequences than those observed during training. Models trained with ALiBi on shorter sequences demonstrate superior perplexity on longer sequences compared to models using traditional positional encodings, while also achieving substantial savings in training time and memory.

Key insights

• Attention with Linear Biases (ALiBi) enables transformers to extrapolate robustly to significantly longer input sequences than those used during training, addressing a core limitation of standard positional encoding methods.
• The benefits of ALiBi in perplexity improvement for longer sequences primarily stem from mitigating the 'early token curse,' where initial tokens in a sequence lack sufficient context for accurate prediction.
• ALiBi achieves superior extrapolation and performance without adding learned parameters or incurring computational overhead, making it highly efficient and practical for large-scale language models.

Summary by alphaXiv.