Reproduce “From promise to practice: realizing high-performance decentralized training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zesen Wang, Jiaojiao Zhang, Xuyang Wu, Mikael Johansson
Researchers from KTH Royal Institute of Technology and Southern University of Science and Technology present a framework for high-performance decentralized training that matches or exceeds All-Reduce performance in diverse computing environments. Their approach introduces a decentralized Adam optimizer, a runtime model, and system designs that enable communication-computation overlap and resilience to stragglers, achieving up to 60% per-iteration speedups for Neural Machine Translation and a 6.31% total training time reduction for GPT-2 pre-training.
Key insights
- Decoupling local model updates from neighbor information enables significant communication-computation overlap, reducing per-iteration runtime.
- The decentralized training scheme inherently provides greater resilience to stragglers compared to synchronous All-Reduce, improving strong scaling efficiency.
- Accumulation of gradients within a decentralized Adam optimizer (AccumAdam) effectively mitigates variance from small local batch sizes, improving generalization performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.