Reproduce “A Survey of End-to-End Modeling for Distributed DNN Training: Workloads, Simulators, and TCO” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jonas Svedas, Hannah Watson, Nathan Laubeuf, Diksha Moolchandani, Abubakr Nada, Arjun Singh, Dwaipayan Biswas, James Myers and 1 more
This paper presents the first comprehensive survey that bridges workload representation, simulation infrastructure, and economic/environmental cost modeling for distributed Deep Neural Network training. It identifies critical challenges such as ecosystem fragmentation in workload representations, limitations in simulator validation at scale, and significant gaps in integrating cost and environmental impact assessments with performance simulations.
Key insights
- Workload representation is fragmented, with many custom intermediate representations (IRs) developed instead of adopting standards, and often omits crucial system-level events like I/O or fault recovery.
- Simulator validation remains a persistent challenge, with most tools validated only at small scales and a heavy bias toward NVIDIA GPUs and simplified network models.
- Economic and environmental cost modeling faces severe data scarcity and is largely isolated from dynamic simulators, preventing holistic evaluation of future system designs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.