Reproduce “FedPop: Federated Population-based Hyperparameter Tuning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Haokun Chen, Denis Krompass, Jindong Gu, Volker Tresp

FedPop introduces an online, population-based evolutionary algorithm for hyperparameter tuning in Federated Learning, efficiently optimizing both client and server-side hyperparameters during training. The approach, which utilizes a two-level tuning mechanism, significantly improves global model accuracy by up to 4.79% on Non-IID CIFAR-10 compared to existing methods and scales effectively to full-sized ImageNet-1K.

Key insights

  • An "online tuning-while-training" paradigm drastically reduces the computational and communication overheads associated with hyperparameter tuning in Federated Learning by integrating optimization with ongoing model training.
  • A two-level evolutionary optimization approach, combining global inter-configuration tuning and local intra-configuration fine-tuning, effectively optimizes diverse hyperparameters and enhances robustness to client data variations.
  • Aggregating validation losses over multiple communication rounds, especially with weighted sums, can mitigate bias introduced by distributed and potentially Non-IID validation data in Federated Learning environments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.