Reproduce “Disentangling the Expressivity of RoPE” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Selim Jerad, Anej Svete, Jiaoda Li, Ryan Cotterell
This research formally disentangles the expressivity of Rotary Position Embeddings (RoPE) within finite-precision transformers, distinguishing that engineered component-periodic RoPE enables arbitrary-length modular reasoning, while conventional non-periodic RoPE provides a bounded simulation of fixed-offset look-back. The work demonstrates how these distinct operational regimes reconcile previously conflicting accounts of RoPE's capabilities.
Key insights
- The expressivity of RoPE hinges on whether its rotational components are perfectly periodic (RoPE$_P$) or non-periodic (RoPE$_{NP}$), with each supporting distinct classes of formal languages.
- While component-periodic RoPE (RoPE$_P$) and periodic Sinusoidal PEs (SiPE$_P$) share theoretical expressivity (LTL[P, MOD]), RoPE$_P$ is empirically more learnable for modular tasks.
- Conventional RoPE (RoPE$_{NP}$) offers a bounded capability for fixed-offset attention but can disrupt global, position-invariant reasoning required for other types of past-operator (P-style) tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.