Reproduce “Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding” on OpenResearch

About this paper

Liang Zhao, Xiachong Feng, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin and 1 more

A comprehensive survey systematically organizes research on length extrapolation in Transformer models, analyzing how various positional encoding designs and recent techniques like position interpolation enable these models to process sequences longer than their training data. It categorizes existing methods from absolute and relative positional encodings to LLM-era advancements, highlighting their mechanisms and performance trade-offs.

Key insights

• Positional encoding is the primary architectural factor determining a Transformer's ability to extrapolate to sequence lengths beyond its training distribution.
• Relative Positional Encodings (RPEs), particularly from the RoPE and T5-Bias families, generally offer superior length extrapolation capabilities compared to Absolute Positional Encodings (APEs).
• Position interpolation and randomized positional encoding are highly effective, low-overhead methods for extending the context windows of pre-trained LLMs, making them practically appealing solutions.
• Current evaluation metrics, such as perplexity, are often insufficient to thoroughly assess length extrapolation on complex downstream tasks, underscoring the need for more robust benchmarks.

Summary by alphaXiv.