Reproduce “Gradient Descent Happens in a Tiny Subspace” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guy Gur-Ari, Daniel A. Roberts, Ethan Dyer
The authors track gradient projections onto leading Hessian eigenspaces and measure how those subspaces persist. In a two-hidden-layer MNIST network, after 50 steps, the top 10 directions contain 0.93 of the squared gradient norm, versus 0.01 for the next 10; initially the top 20 contain 0.20. Across three classifier settings, leading subspaces persist more than next-direction groups, suggesting concentration, not confinement.
Key insights
- The gradient of the loss during training rapidly aligns itself within a small subspace spanned by the top k Hessian eigenvectors, where k is the number of classes.
- This top k-dimensional Hessian subspace, characterized by positive curvature, remains approximately preserved and stable over long training durations.
- The observed dynamics suggest that deep learning optimization effectively occurs within a low-dimensional, pseudo-convex region, potentially explaining the effectiveness of gradient descent.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.