Reproduce “HetCCL: Accelerating LLM Training with Heterogeneous GPUs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Heehoon Kim, Jaehwan Lee, Taejeoung Kim, Jongwon Park, Jinpyo Kim, Pyongwon Suh, Ryan H. Choi, Sangwoo Lee and 1 more
HetCCL is a collective communication library designed to accelerate large language model training on GPU clusters comprising both NVIDIA and AMD hardware. It achieves up to 1.48x faster training compared to NVIDIA-only configurations and up to 2.97x faster when compared to AMD-only setups by enabling unified utilization of diverse GPU resources.
Key insights
- RDMA provides an effective mechanism for point-to-point communication between GPUs from different vendors without requiring low-level driver modifications, abstracting away hardware specificities.
- Decoupling runtime API calls and device-dependent kernel code into an abstraction layer and dynamically loaded libraries enables cross-vendor execution from a single compiled binary.
- By orchestrating vendor-specific collective libraries for local operations while managing cross-vendor data movement, HetCCL maintains near-native performance while extending functionality to heterogeneous clusters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.