Reproduce “Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Graphs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Da Zheng, Xiang Song, Chengru Yang, Dominique LaSalle, George Karypis
DistDGLv2 introduces a distributed system for training Graph Neural Networks on billion-scale heterogeneous graphs utilizing hybrid CPU/GPU clusters. The system achieves 2-3x speedup over existing GPU-adapted distributed GNN frameworks and up to 15x over CPU-only counterparts while preserving model accuracy through optimized data locality, load balancing, and asynchronous processing.
Key insights
- Asynchronously pipelining mini-batch generation stages effectively hides communication and CPU overheads, which is particularly beneficial for GNNs characterized by lower computation density.
- Employing a two-level graph partitioning scheme at both the machine and GPU levels substantially improves data locality and reduces inter-device data movement during training.
- Maintaining synchronous Stochastic Gradient Descent (SGD) with ego-networks that include non-local vertices is crucial for ensuring unbiased gradient estimation and achieving high model accuracy in a distributed setting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.