Reproduce “Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi, Karol Pajak, James Snewin, Harry Xi, Rodney O'Donnell, Thalaiyasingam Ajanthan and 4 more
Agora trains an 8.6B model across internet-connected GPU hosts by splitting it into stages, compressing stage-to-stage activations by about 100×, and averaging rotating weight subsets asynchronously. Pluralis-8B processed 500B tokens over 40 days with 330 contributor nodes joining and leaving. It reached loss 2.41 after 60,000 steps; its loss tracked a centralized reference only through 27,000 iterations.
Key insights
- Efficient internet-scale LLM training is achievable by implementing communication-efficient pipeline parallelism and fault-tolerant asynchronous data parallelism, shifting the bottleneck from network to computation.
- The system can maintain training stability and convergence despite continuous node churn, hardware heterogeneity, and intermittent communication failures, demonstrating robust resilience.
- Network latency, rather than raw bandwidth (beyond a minimum threshold), is the primary limiting factor for per-worker throughput in such highly distributed environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.