Reproduce “Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks” on OpenResearch

About this paper

Mansheej Paul, Brett W. Larsen, Surya Ganguli, Jonathan Frankle, Gintare Karolina Dziugaite

This research explored the data and duration needed for the critical pre-training phase in Iterative Magnitude Pruning (IMP), demonstrating that small subsets of "easy" data can efficiently yield sparse networks with performance comparable to using full datasets, and identified that per-example linear mode connectivity serves as a stronger predictor of sparse network success than dense network accuracy.

Key insights

• For IMP pre-training, smaller subsets of carefully selected data (particularly "easy" examples) can be as effective or more efficient than using the full dataset, challenging the 'more data' assumption for this specific phase.
• The geometry of the loss landscape, specifically the distribution of per-example linear mode connectivity, provides a better indicator of an effective IMP initialization than the dense network's pre-training accuracy.
• The data requirements for stabilizing training via learning rate warmup differ from those for finding a good IMP initialization, suggesting distinct roles for data in different early training dynamics.

Summary by alphaXiv.