Reproduce “Relative Positions Generalize, Absolute Positions Memorize: An Implicit-Bias Account of Length Generalization in Attention” on OpenResearch

About this paper

Subham Singh, Ashutosh Mishra, Subha Raut

This work provides an implicit-bias account for length generalization differences in Transformers, demonstrating that Rotary Positional Embeddings (RoPE) enable gradient descent to select length-invariant attention rules via offset-equivariance and a "target-aligned carrier," while learned Absolute Positional Encodings (APE) fail due to "position pinning." RoPE-based models exhibit graceful accuracy decay at longer lengths, which is explained by an "attention dilution" law.

Key insights

• RoPE ensures that the attention logit depends solely on the relative offset, leading to exact equivariance and allowing the learned rule to transfer identically to all available lengths.
• Gradient descent implicitly selects a "target-aligned carrier" in RoPE models, where content vectors in query and key spaces align to maximize the attention score at the target relative offset.
• Learned Absolute Positional Encodings (APE) fail because position embeddings beyond the maximum training length receive no gradients, leading to "position pinning" and an inability to address unseen positions.
• Softmax normalization plays a critical role in inducing the target-aligned carrier and enabling extrapolation, as linear attention models (without softmax) fail to generalize.

Summary by alphaXiv.