Reproduce “Improving Convergence and Generalization Using Parameter Symmetries” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bo Zhao, Robert M. Gower, Robin Walters, Rose Yu

The authors introduce teleportation, searching equivalent parameter settings for larger gradients that alter later updates while preserving loss under exact symmetries. Their theory bounds the minimum expected maximum squared gradient norm across the symmetry orbit by a quantity decreasing as the inverse square root of training steps. CIFAR-10 experiments associate curvature interventions with validation loss locally, without establishing causation.

Key insights

  • Symmetry teleportation intrinsically introduces a quadratically contracting term, establishing a theoretical link to dampened Newton steps and suggesting a fundamental mechanism for accelerated convergence.
  • The 'curvature of minima' (curvature of the loss-invariant manifold) serves as a novel geometric property correlated with generalization, indicating that minima with larger curvature may lead to better model performance.
  • Parameter space symmetries, when strategically exploited through teleportation, can be leveraged to not only accelerate optimization but also to actively search for parameter configurations that improve generalization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.