Reproduce “MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Juntao Zhao, Qi Lu, Wei Jia, Borui Wan, Lei Zuo, Junda Feng, Jianyu Jiang, Yangrui Chen and 8 more
MegaScale-Data, a distributed data loading architecture, significantly improves large foundation model training by achieving up to a 3.63x speedup in iteration time and a 13.5x reduction in loader memory usage. This system optimizes for multisource data, heterogeneous preprocessing, and hybrid parallelism by eliminating data redundancies and enabling adaptive resource management.
Key insights
- Disaggregating data processing roles into Source Loaders and Data Constructors effectively eliminates parallelism and source-level data redundancies, leading to substantial memory and throughput gains.
- Centralized and declarative data orchestration allows for flexible, cost-aware balancing strategies that dynamically adapt to heterogeneous data and evolving mixing ratios, mitigating workload imbalances.
- The actor-model based architecture, particularly the Data Constructor, proves essential for scalable data delivery, preventing communication bottlenecks that emerge at extreme GPU counts (2,000 to 4,000 GPUs).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.