Reproduce “On the saddle point problem for non-convex optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Razvan Pascanu, Yann N. Dauphin, Surya Ganguli, Yoshua Bengio
Pascanu et al. introduce the "saddle-free Newton method," an optimization algorithm designed to efficiently navigate and escape saddle points in high-dimensional non-convex error landscapes. This method, which leverages the inverse of the absolute Hessian to rescale gradients, achieves lower training error and faster convergence compared to standard gradient descent and damped Newton methods on a neural network model, consistently reducing the magnitude of negative eigenvalues.
Key insights
- Saddle points, not high-error local minima, pose the primary impediment to efficient optimization in high-dimensional non-convex landscapes.
- The optimal step near a critical point involves scaling the gradient by the inverse of the absolute eigenvalue for each eigenvector direction, providing accelerated movement while correctly escaping saddle points.
- A generalized trust region formulation with a constraint based on the absolute value of the second-order Taylor expansion term can lead to the desired saddle-free Newton update.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.