Reproduce “Direct Model State Migration for Elastic Training of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weijian Liu, Mingzhen Li, Rui Kang, Chen Sun, Guangming Tan, Weile Jia
ETC reduces state-migration time when training jobs change GPU assignments and hybrid-parallel layouts. It matches old and new workloads to limit movement, then transfers state directly between GPUs while retaining resident state. For GPT3-32B reconfigured from PP8TP1 to PP16TP1 on Cluster-A, ETC reports 1,500.88 ms versus 4,357.24 ms for a composite checkpoint baseline; these are migration-phase times, not total resumption time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.