Reproduce “FedMuon: Federated Learning with Bias-corrected LMO-based Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuki Takezawa, Anastasia Koloskova, Xiaowen Jiang, Sebastian U. Stich
Researchers from Kyoto University, OIST, University of Zurich, and CISPA developed FEDMUON, a federated learning algorithm that integrates the Muon optimizer by correcting a fundamental bias of the Linear Minimization Oracle (LMO) operator. The method provides theoretical convergence guarantees even with inexact LMO solutions and empirically outperforms state-of-the-art adaptive federated learning baselines on diverse benchmarks.
Key insights
- The Linear Minimization Oracle (LMO) operator is fundamentally biased in federated settings due to its non-linearity, causing naive integrations (LOCALMUON) to fail convergence in heterogeneous environments.
- Bias correction through control variates successfully mitigates this LMO bias, allowing federated LMO-based optimizers to provably converge to stationary points.
- FEDMUON maintains convergence even when the LMO is solved approximately (e.g., with any number of Newton-Schulz iterations T >= 0), with increasing LMO accuracy (higher T) leading to convergence in stronger Schatten norms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.