Reproduce “DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuanqing Wang, Yuchen Zhang, Hao Lin, Junhao Hu, Chunyang Zhu, Quanlu Zhang, Boxun Li, Guohao Dai and 4 more
DynaTrain lets training jobs change parallelism during execution by routing model state in memory between GPUs. Its Virtual Parameter Space maps differently sharded parameters, gradients, and optimizer state to shared logical coordinates, so ranks can identify what to send, retain, or receive. On the reported Qwen3-235B-A22B transition, it took 4.36 seconds versus 1,456.39 seconds for checkpoint save and load, a 334.03× speedup; this result is specific to tested transitions on A800s.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.