Reproduce “Symmetry Teleportation for Accelerated Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bo Zhao, Nima Dehmamy, Robin Walters, Rose Yu

An optimization technique leverages parameter space symmetries to accelerate convergence in deep learning. It enables "teleportation" along loss level sets to points with steeper gradients, reducing epochs and wall-clock time across various tasks and showing a theoretical link to second-order methods.

Key insights

  • Teleportation accelerates the rate of loss decay by strategically moving parameters to points on the same loss level set with larger gradient magnitudes.
  • For convex quadratic functions, maximizing gradient magnitude on a loss level set is equivalent to minimizing the Euclidean distance to the global minimum and ensures sustained optimality of subsequent gradient flow.
  • At points where the gradient norm is locally maximized on a loss level set, the gradient direction aligns with Newton's method, suggesting a way to gain second-order benefits via first-order computations within symmetry orbits.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.