Reproduce “LiveR: Fine-Grained Elasticity via Live Reconfiguration for Model Training” on OpenResearch

About this paper

Haoyuan Liu, Kairui Zhou, Shuyao Qi, Qinwei Yang, Shengkai Lin, Shizhen Zhao, Wei Zhang

The LiveR system, from researchers at Shanghai Jiao Tong University and the University of Connecticut, enables fine-grained elasticity for large language model training by facilitating live reconfiguration, reducing scaling event downtime from minutes to seconds, which is a 14x to 23x speedup compared to checkpoint/restart baselines. This approach allows efficient utilization of volatile GPU resources by sustaining over 99% training goodput under high resource volatility.

Key insights

• Decoupling the setup and initialization of a new distributed training topology (Shadow World) from the active training loop (Active World) significantly reduces critical path downtime.
• "Mock Process Groups" enable new or reconfigured ranks to perform heavy local setup and warm-up without stalling global collective operations of the active training process.
• Arbitrary reshaping of multi-dimensional parallelism (TP/PP/DP) can be achieved with bounded memory by streaming tensor shards peer-to-peer through a fixed-size staging buffer, avoiding full model materialization.

Summary by alphaXiv.