Reproduce “Momentum Benefits Non-IID Federated Learning Simply and Provably” on OpenResearch

About this paper

Ziheng Cheng, Xinmeng Huang, Pengfei Wu, Kun Yuan

Researchers from Peking University and the University of Pennsylvania introduced momentum-enhanced variants of FEDAVG and SCAFFOLD, which are proven to converge without the restrictive assumption of bounded data heterogeneity. These algorithms achieve state-of-the-art convergence rates and significantly accelerate training across various non-IID settings and partial client participation scenarios.

Key insights

• Momentum can enable FEDAVG-type algorithms to converge provably without the assumption of bounded data heterogeneity, a first-of-its-kind theoretical result for non-convex FL.
• SCAFFOLD-M achieves unconditionally faster convergence than vanilla SCAFFOLD, demonstrating that momentum provides inherent benefits even for robust algorithms.
• The proposed momentum-enhanced variance-reduced algorithms (FEDAVG-M-VR, SCAFFOLD-M-VR) are the first to combine variance reduction with convergence guarantees under partial participation and unbounded data heterogeneity.

Summary by alphaXiv.