Reproduce “Towards Sparsification of Graph Neural Networks” on OpenResearch

About this paper

Hongwu Peng, Deniz Gurevin, Shaoyi Huang, Tong Geng, Weiwen Jiang, Omer Khan, Caiwen Ding

This research explores two state-of-the-art model sparsification techniques for Graph Neural Networks, demonstrating that dynamic sparse training can reduce training FLOPs by over 8 times compared to train-and-prune methods while maintaining or improving model accuracy. The study from the University of Connecticut, University of Rochester, and George Mason University is among the first to apply dynamic sparse training to GNNs, addressing the computational demands of large-scale graph learning.

Key insights

• Dynamic sparse training offers substantial computational efficiency benefits (e.g., >8x FLOPs reduction) for GNNs compared to train-and-prune methods, operating with fixed sparsity from the outset.
• Sparsity in GNN weights can act as a form of regularization, sometimes leading to improved accuracy, particularly under extreme sparsity levels.
• Sparsifying input node embeddings significantly reduces FLOPs for GNN models where the first layer dominates computation, with minimal impact on accuracy.

Summary by alphaXiv.