Reproduce “CrossPipe: Towards Optimal Pipeline Schedules for Cross-Datacenter Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tiancheng Chen, Ales Kubicek, Langwen Huang, Torsten Hoefler
CrossPipe from ETH Zürich develops a framework to optimize pipeline schedules for Large Language Model training distributed across multiple datacenters, explicitly accounting for high latency and limited bandwidth. The system achieves up to a 33.6% reduction in training runtime compared to static schedules and demonstrates that cross-datacenter Pipeline Parallelism can provide a 3.05x speedup over Data Parallelism under bandwidth limitations.
Key insights
- Cross-datacenter Pipeline Parallelism (PP) generally provides superior efficiency compared to cross-datacenter Data Parallelism (DP) for large LLMs, especially under bandwidth constraints, due to its communication volume scaling.
- Dynamic pipeline schedules generated by CrossPipe, particularly those employing finer-grained block splitting (e.g., DGrad and WGrad), significantly reduce pipeline bubbles and improve performance compared to static schedules in high-delay cross-datacenter environments.
- Both latency and bandwidth contribute to runtime delays, but bandwidth can become a critical bottleneck when its delay component exceeds the per-microbatch forward computation time, leading to additional queuing-induced pipeline bubbles.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.