Reproduce “On Provable Benefits of Muon in Federated Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinwen Zhang, Hongchang Gao
Researchers at Temple University developed FedMuon, a federated learning algorithm that integrates the Muon optimizer to enhance distributed training. The method provides theoretical convergence guarantees with hyperparameters independent of problem-specific constants, naturally accommodates heavy-tailed noise, and empirically achieves superior performance over baselines on diverse benchmarks.
Key insights
- FedMuon's optimal hyperparameters (learning rate, momentum, communication period) are theoretically derived to be independent of problem-specific parameters, simplifying deployment and tuning.
- The inherent orthonormalization operation of Muon allows FedMuon to naturally accommodate heavy-tailed noise without requiring explicit gradient clipping, enhancing robustness.
- Theoretical analysis demonstrates that FedMuon achieves a linear speedup with respect to the number of workers (K) in its convergence rates under both bounded variance and heavy-tailed noise conditions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.