Reproduce “RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodings” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jarod Lévy, Mathurin Videau, Jad Yehya, Jean-Rémi King, Stéphane d'Ascoli, Thomas Moreau
DaRoPE introduces a hybrid positional encoding that preserves ordinary positions on fast frequency bands while assigning bounded, learned, context-dependent coordinates to slow bands. In a synthetic return-from-digression probe after a 2,048-token digression, a 1B model selects the earlier correct answer 88.1% of the time, versus 77.6% for RoPE-500k. This tests retrieval under interference, not ordinary long-document performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.