Reproduce “DynMuon: A Dynamic Spectral Shaping View of Muon” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fangzhou Wu, Rikhav Shah, Sandeep Silwal, Qiuyi Zhang
DynMuon introduces a dynamic spectral shaping optimization strategy for large language models, moving beyond fixed spectral transformations by adaptively adjusting a spectral exponent throughout training. This method consistently achieves lower validation loss and improves convergence speed by 10.6% to 26.5% compared to the Muon optimizer, with negligible per-step computational cost.
Key insights
- The residual error signal in deep neural networks progressively concentrates in "flat," low-curvature modes during the latter stages of training.
- A critical signal-noise trade-off exists where decreasing the spectral exponent `p` can improve residual signal contraction in flat modes but also amplify stochastic noise in those same directions.
- Optimal spectral shaping is dynamic; positive exponents are beneficial early to accelerate strong-mode contraction, while mildly negative exponents help late in training by addressing residual signal in flat modes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.