Reproduce “How many degrees of freedom do we need to train deep networks: a loss landscape perspective” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Brett W. Larsen, Stanislav Fort, Nic Becker, Surya Ganguli
Researchers from Stanford University developed a theoretical framework to explain the empirically observed threshold training dimension required for deep network optimization within low-dimensional random subspaces, revealing its dependence on initialization quality. Their work introduced "lottery subspaces" that achieve high accuracy with fewer training dimensions than random subspaces or traditional lottery tickets.
Key insights
- The existence of a sharp threshold training dimension is explained by the relationship between the number of available dimensions and the "local angular dimension" of the loss sublevel set around the initialization point.
- Starting training from an "informed" initialization, achieved through a "burn-in" phase, significantly reduces the required training dimensions by effectively enlarging the local angular extent of the target loss region.
- The high-dimensional geometry of the loss landscape, particularly the "Gaussian width" of projected loss sublevel sets, dictates how easily an optimizer can find low-loss regions within a constrained subspace.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.