Reproduce “TrasMuon: Trust-Region Adaptive Scaling for Orthogonalized Momentum Optimizers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peng Cheng, Jiucheng Zang, Qingnan Li, Liheng Ma, Yufei Cui, Yingxue Zhang, Boxing Chen, Ming Jian and 1 more
TrasMuon, an optimizer from researchers at Huawei, Waterloo, and McGill, integrates orthogonalized momentum updates with adaptive trust-region scaling to enhance deep neural network training. It delivers faster early-stage convergence, exemplified by reaching a training loss of 7.0 in 80 steps (vs. AdamW's 188 steps), and improved stability, particularly in warmup-free settings, across various large-scale models.
Key insights
- A more robust and efficient optimization algorithm can be achieved by explicitly combining structured geometric updates (orthogonalization) with adaptive magnitude control, addressing both global step-size calibration and localized energy bursts.
- Employing a robust reference for typical column energy (e.g., median) allows for effective, adaptive damping of localized gradient spikes without being overly sensitive to outliers, preserving overall optimization stability.
- Integrating schedule-free averaging for the damping signal, along with RMS calibration, can significantly reduce the optimizer's dependence on delicate learning rate schedules and long warmup periods, enhancing practical applicability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.