Reproduce “H2:Towards Efficient Large-Scale LLM Training on Hyper-Heterogeneous Cluster over 1,000 Chips” on OpenResearch

About this paper

Ding Tang, Jiecheng Zhou, Jiakai Hu, Shengwei Li, Huihuang Zheng, Zhilin Pei, Hui Wang, Xingcheng Zhang

The H2 framework, developed by researchers at the Shanghai Artificial Intelligence Laboratory, enables efficient training of 100-billion-parameter large language models on hyper-heterogeneous clusters containing over 1,000 chips from multiple different architectures. It achieves superlinear speedup in some configurations by unifying software interfaces, optimizing inter-chip communication, and adaptively managing parallelism to leverage diverse hardware capabilities.

Key insights

• Overcoming software and communication isolation in hyper-heterogeneous clusters requires a unified operator interface (DIOPI) for consistent semantics and device-direct RDMA communication (DiComm) for efficient inter-vendor data transfer.
• Optimal performance in diverse hardware environments can be achieved by an adaptive pipeline parallelism strategy (HeteroPP) that leverages complementary chip strengths, such as memory-aware allocation and non-uniform layer distribution.
• Efficient automated strategy search (HeteroAuto) can explore vast configuration spaces for heterogeneous parallelization with minimal overhead, identifying near-optimal solutions faster than prior methods.

Summary by alphaXiv.