Reproduce “The Early Phase of Neural Network Training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jonathan Frankle, David J. Schwab, Ari S. Morcos

The authors test whether early training changes support trainable sparse subnetworks in ResNet-20 on CIFAR-10. Using iterative magnitude pruning with rewinding, they repeatedly prune low-magnitude weights and reset survivors to earlier checkpoints. Rewinding to 500 iterations or later lets subnetworks match the full model’s 91.5 ± 0.2% evaluation accuracy with 16.8% of weights remaining; gains plateau after 2,000 iterations in these experiments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.