Reproduce “Rotated Manifold Optimization for Low-Rank Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhui Ding, Javier Zazo, James Hensman
RoM optimizes LoRA’s low-rank weight update directly, rather than its redundant factorization. It projects gradients and momentum onto the fixed-rank manifold, then takes normalized steps in a rotated frame. It achieves the lowest validation loss in math SFT, but downstream accuracy leads are small; in one RL setup, it beats Adam on three held-out benchmarks, with 1.08× per-step time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.