Reproduce “EROICA: Online Performance Troubleshooting for Large-scale Model Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu Guan, Zhiyu Yin, Haoyu Chen, Sheng Cheng, Chaojie Yang, Kun Qian, Tianyin Xu, Pengcheng Zhang and 6 more
EROICA, an online performance troubleshooting system from Alibaba Cloud and the University of Illinois Urbana-Champaign, integrates fine-grained observability with cluster-wide coverage to diagnose issues in Large Model Training (LMT). It successfully identified root causes for 78 out of 80 severe LMT performance problems, improving training throughput by 20-100%.
Key insights
- LMT performance problems can be effectively diagnosed by analyzing deviations in function runtime behavior patterns on the critical path, even across thousands of GPUs.
- Conditional activation of fine-grained profiling allows detailed observation of issues only when they occur, circumventing the high overhead of continuous full-stack monitoring.
- Representing function behavior with timestamp-independent metrics enables robust pattern comparison across distributed workers without stringent clock synchronization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.