Reproduce “Robust Pruning at Initialization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Soufiane Hayou, Jean-Francois Ton, Arnaud Doucet, Yee Whye Teh

The authors analyze pre-training pruning and propose scaling to preserve trainability. Sensitivity-Based Pruning ranks initialized weights by the magnitude of each weight times its loss gradient. Feedforward and convolutional networks are rescaled toward the Edge of Chaos while residual networks use Stable ResNet. On CIFAR-10, ResNet104 at 99.5% sparsity reached 87.47 ± 0.23% accuracy, versus 33.63 ± 33.27% for SNIP.

Key insights

  • Initialization on the "Edge of Chaos" (EOC) is necessary for effective Sensitivity-Based Pruning (SBP) in FFNNs and CNNs, and pruning itself can disrupt these critical conditions.
  • Residual Networks are inherently well-conditioned for pruning at initialization, maintaining EOC properties even when sparse, but standard ResNets exhibit an exploding gradient problem for the pruning criterion with increasing depth.
  • Simple post-pruning weight rescaling for FFNNs/CNNs and a specific re-parameterization for ResNets can effectively restore stable signal propagation, enhancing the trainability of deep sparse networks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.