Reproduce “Picking Winning Tickets Before Training by Preserving Gradient Flow” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chaoqi Wang, Guodong Zhang, Roger Grosse

Researchers at the University of Toronto and the Vector Institute developed Gradient Signal Preservation (GraSP), a method for pruning deep neural networks at initialization to identify trainable sparse sub-networks. GraSP prunes weights based on their impact on the network's gradient flow, achieving superior performance at high sparsity levels, for instance, outperforming SNIP by 6.2% top-1 accuracy on ResNet-50 at 90% sparsity on ImageNet.

Key insights

  • At initialization, preserving the training dynamics, specifically the flow of gradients, is more critical for effective sparse network training than merely preserving the initial loss value.
  • Evaluating connection importance in isolation, as done by SNIP, can create bottlenecks that severely impede information and gradient propagation through the network.
  • GraSP's approach of preserving the gradient norm implicitly encourages a favorable structure in the Neural Tangent Kernel (NTK), leading to more efficient optimization of the sparse sub-network.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.