Reproduce “Linear Mode Connectivity and the Lottery Ticket Hypothesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin
This study by Frankle et al. introduces instability analysis to demonstrate that neural networks become stable to stochastic gradient descent noise early in training. This early stability enables Iterative Magnitude Pruning (IMP) to find sparse subnetworks that match the full network's performance when rewound to these stable intermediate states, resolving previous inconsistencies for larger models.
Key insights
- Neural network training progresses through an initial 'unstable' phase, where SGD noise significantly affects the final minimum, followed by a 'stable' phase where optimization converges to a linearly connected region.
- The performance of an IMP-identified sparse subnetwork is directly linked to its stability to SGD noise; only stable subnetworks achieve matching accuracy.
- Matching subnetworks can be found for complex models by rewinding to early training steps (k > 0), even if they are not present at initialization (k=0).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.