Reproduce “Rotary Orientation Contraction in RoPE” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Andrew Mouldon, Eren Gultepe
ROC contracts each two-coordinate query and key pair’s pre-RoPE orientation toward the positive horizontal axis before the usual rotary position embedding rotation, leaving positional frequencies unchanged. With QK normalization, its 124M-parameter model reaches FineWeb-Edu validation perplexity 19.09 ± 0.01, versus 19.31 ± 0.02 for RoPE across three matched seeds. The advantage was absent without QK normalization; evidence covers two scales and one domain.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.