Reproduce “Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong, Tuc Nguyen, James Bailey, Trung Le
Spectral analysis explains why Muon can tolerate larger learning rates and converge faster than SGD. By replacing gradient singular values with a common scale while preserving singular directions, Muon shifts the stability constraint toward the average singular value and improves an effective curvature ratio under a K-FAC model; CIFAR-10 and transformer experiments support these predictions, including improved stability with Frobenius-based normalization.
Key insights
- Orthogonalization limits the disproportionate influence of a dominant gradient singular direction, allowing a larger predicted step size than SGD under the same local curvature.
- Muon’s preconditioner reduces spectral anisotropy in the gradient-related curvature term, which theoretically yields a better linear convergence factor even at an equal learning rate.
- The analysis links normalization to optimization stability through the spectrum of input covariance, rather than only through coordinate-wise activation magnitudes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.