Reproduce “Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chamin Hewa Koneputugodage, Thalaiyasingam Ajanthan, Sameera Ramasinghe, Hadi Mohaghegh Dolatabadi, Shamane Siriwardhana, Gil Avraham, Violetta Shevchenko, Karol Pajak and 2 more

The authors introduce a language-model training method that overlaps synchronization with local work and optionally adds blocking pairwise mixing. In an eight-worker, 1.5B-parameter Llama 3 experiment with 100 inner steps, it reaches 66% simulated utilization at 200 Mbps versus DiLoCo’s 53%, with validation perplexity of 15.60 versus 15.52 at 30B tokens; evidence is simulator-based and limited to 1.5B parameters.

Key insights

  • Factoring synchronization into non-blocking parameter mixing (Mix1) and selectively blocking gradient mixing (Mix2) significantly improves compute utilization by overlapping communication with local computation.
  • The Jensen–Shannon (JS) distance effectively tracks functional disagreement and optimization instability, proving more informative than L2 distance for guiding selective blocking communication on sensitive parameter blocks.
  • Replacing global all-reduce with gossip-based mixing enhances robustness to communication failures, enabling graceful degradation rather than full process abortion.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.