Reproduce “A Quadratic Lens on Muon's Polar Update: Benefits, Pitfalls, and the Role of Newton-Schulz” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Egor Shulgin, Sam Laing, Antonio Orvieto, Peter Richtárik
The authors analyze PolarGD, the idealized orthogonalized update at Muon’s core, and finite Newton–Schulz approximations on quadratic objectives. A controlled near-rank-deficient quadratic shows finite approximations converge faster than exact PolarGD by damping tiny singular modes. In a trained 90M-parameter transformer, about a quarter of attention-matrix momentum singular values fall below the five-step damping threshold; this diagnostic does not establish faster training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.