Reproduce “Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
John Thorpe, Pengzhan Zhao, Jonathan Eyolfson, Yifan Qiao, Zhihao Jia, Minjia Zhang, Ravi Netravali, Guoqing Harry Xu
Bamboo, developed by researchers from UCLA, CMU, Microsoft Research, and Princeton, introduces a resilient distributed training system that enables cost-effective training of large Deep Neural Networks (DNNs) on unstable preemptible cloud instances. The system leverages redundant computation within pipeline parallelism, exploiting idle GPU cycles to hide redundancy overhead, and achieves a 3.6x cost reduction compared to on-demand instances while maintaining high throughput.
Key insights
- Pipeline bubbles can be strategically repurposed to hide the overhead of redundant forward computations, making resilience nearly free during normal operation.
- A decoupled "eager forward, lazy backward" redundancy strategy effectively balances low runtime overhead with rapid fault recovery in dynamic cloud environments.
- Localizing redundant computations to adjacent nodes in a pipeline minimizes communication overhead, preserving efficiency during both normal operation and failover.
- Memory management via swapping intermediate redundant results to CPU memory is crucial for accommodating redundancy in memory-intensive large DNN training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.