Reproduce “nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Boyang Li, Yulin Wu, Sizhe Xu, Nuoxian Huang, Zhonghang Yuan, Shangyi Guo, Shu Yang, Takahiro Yabe
nD-RoPE, developed by researchers from New York University, Peking University, Imperial College London, and USTC, generalizes Rotary Position Embedding to arbitrary n-dimensional Euclidean spaces. It employs a multi-scale, simplex-based wave-vector design to achieve isotropic spatial representations, leading to improved in-domain performance, enhanced rotational robustness, and better generalization across diverse high-dimensional vision tasks.
Key insights
- A principled generalization of RoPE to n dimensions can be achieved by treating positions and wave vectors as unified n-dimensional entities, derived from Fourier-based translation-invariant properties.
- Adopting a regular simplex for wave-vector configuration ensures maximum geometric symmetry and isotropic frequency coverage, which is crucial for mitigating directional biases and enhancing rotational robustness.
- Effective application in high-dimensional settings benefits from a multi-scale approach to frequency distribution, with a balanced allocation of frequencies across attention heads optimizing performance and generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.