Reproduce “Why Are Positional Encodings Nonessential for Deep Autoregressive Transformers? Revisiting a Petroglyph” on OpenResearch

About this paper

Kazuki Irie

A paper from Harvard University clarifies that explicit positional encodings are not essential for multi-layer autoregressive Transformer language models. The work re-establishes an understanding from the pre-LLM era that such deep architectures implicitly capture positional information, often leading to improved length generalization.

Key insights

• While standalone self-attention is permutation invariant, a cascade of multiple self-attention layers within an autoregressive Transformer architecture becomes 'fully position-sensitive' without explicit positional encodings.
• The necessity of explicit positional encodings is contingent on the Transformer's depth and operational mode: multi-layer autoregressive models can implicitly learn position, while one-layer autoregressive and all non-autoregressive models require them.
• The causal attention mask inherent in autoregressive processing, combined with network depth, enables the model to infer sequential order information implicitly.

Summary by alphaXiv.