Reproduce “EnergyCal: Taming Loss Spikes in Muon-Family Optimizers for Sparse MoE Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Masato Fujitake
EnergyCal scales routed expert updates in sparse mixture-of-experts training using each post-Newton–Schulz direction’s normalized energy, without changing Muon’s transform or adding optimizer state. In one 4B-parameter model, its MuonH recipe reduced mean thresholded spike burden by 99.51% versus selected MuonH, while final cross-entropy was lower. This three-seed, fixed-token-budget comparison used different peak learning rates, so it does not isolate calibration’s effect.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.