Reproduce “Rethinking the Value of Network Pruning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, Trevor Darrell

For structured network pruning, training the identified architecture from random initialization often matches or exceeds performance achieved by fine-tuning models with inherited weights, indicating that pruning methods primarily derive efficient network structures. The work also offers a re-evaluation of the Lottery Ticket Hypothesis, showing its benefits are contingent on specific training conditions.

Key insights

  • For structured pruning, the specific weights inherited from a large model are often not essential; random initialization and training from scratch can be equally or more effective.
  • Automatic structured pruning methods primarily derive efficient *architectures* rather than selecting "important" weights, suggesting they act as an implicit form of neural architecture search.
  • The advantages observed in the Lottery Ticket Hypothesis might be sensitive to specific training hyperparameters (like learning rate) and the type/scale of pruning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.