Reproduce “Sparse Training from Random Initialization: Aligning Lottery Ticket Masks using Weight Symmetry” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohammed Adnan, Rohan Jain, Ekansh Sharma, Rahul Krishnan, Yani Ioannou
The paper demonstrates that aligning Lottery Ticket Hypothesis (LTH) masks with new random initializations through permutation matching enables effective sparse training from arbitrary initializations. This approach, leveraging insights into weight symmetry and loss landscape geometry, achieves performance comparable to standard LTH while also generating more functionally diverse sparse models for tasks like ensemble learning.
Key insights
- The failure of Lottery Ticket Hypothesis (LTH) masks to transfer to different random initializations is caused by misalignment of optimization basins due to neural network weight symmetry.
- Permuting LTH masks to align them with new initialization basins enables effective sparse training from arbitrary random seeds.
- Sparse solutions obtained via Iterative Magnitude Pruning (IMP) remain linearly connected to their dense counterparts in the loss landscape when accounting for variance collapse.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.