Reproduce “Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoyang Li, Fangcheng Fu, Hao Ge, Sheng Lin, Xuanyu Wang, Jiawen Niu, Yuming Zhou, Xupeng Miao and 1 more
Hetu v2 introduces Hierarchical and Heterogeneous SPMD (HSPMD), an extension to the conventional Single-Program Multiple-Data paradigm that uses declarative annotations for asymmetric sharding and hierarchical communication to efficiently manage spatial and temporal heterogeneity in large-scale deep learning training. The system delivers up to 2.9x speedups over baselines in mixed-hardware environments and enables restart-free reconfigurations with lower overheads for unstable devices and mixed-length data workloads.
Key insights
- Extending the SPMD paradigm at a primitive level with declarative annotations, rather than layering external schedulers, provides native and general support for asymmetric sharding and hierarchical communication.
- Characterizing multifaceted heterogeneity into fundamental spatial and temporal dimensions allows for modular and composable system abstractions that can address diverse scenarios.
- Optimizing inter-device communication through hierarchical resolution and consolidated batching (Fused BSR) is crucial for minimizing overheads in complex heterogeneous distributed settings.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.