Reproduce “An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingjun Zhang, Xiaohe Hu, Menghao Zhang, Ziteng Chen, Yanmin Jia, Yan Zhang, Da Liu, Qing Chen and 16 more
The authors introduce VCCL, a communication library that moves point-to-point coordination from GPU streaming multiprocessors to CPU threads and uses GPU copy engines for necessary transfers. In Megatron-LM training tests of a 314B GPT-2 model, VCCL reported up to 5.28% higher per-GPU training-compute throughput per iteration than NCCL v2.26 at 1,024 GPUs; this measures throughput, not convergence or reliability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.