Reproduce “Hybrid Momentum Methods for Optimizing Deep Learning Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haifeng Xu, Yijin Ren, Chenyu Zhou, Qi Deng, Dongdong Ge, Yinyu Ye

HyM combines a growing long-term gradient memory, which retains weight on early gradients, with a fast state that emphasizes recent gradients. For matrix-valued parameters, HyMUON applies Newton–Schulz orthogonalization to the combined direction. On LLaMA-1.3B pretraining on C4, HyMUON reached validation loss 2.777 versus reproduced Muon’s 2.903 after 10,000 steps and equal token budgets; its settings were tuned.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.