Reproduce “Remove Symmetries to Control Model Expressivity and Improve Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Liu Ziyin, Yizhou Xu, Isaac Chuang

Researchers from Massachusetts Institute of Technology, EPFL, and NTT Research developed `syre`, a model- and symmetry-agnostic algorithm designed to remove symmetries from neural network loss functions. The method provably enhances model expressivity and improves optimization by decoupling symmetric solutions from low-norm states, leading to better performance in various deep learning tasks, including an approximate 50% improvement in linear evaluation accuracy for SimCLR on CIFAR-100.

Key insights

  • Symmetries fundamentally limit model capacity by masking effective features and reducing the effective parameter dimension, causing gradient-based methods to remain confined to lower-dimensional symmetric subspaces.
  • The `syre` algorithm induces a "super-Lipschitz" property, ensuring a strong, non-zero gradient component (`Ω(γσ_0)`) exists to push parameters out of any symmetric solution.
  • By decoupling symmetric solutions from energetically favored low-norm solutions, `syre` effectively breaks the mechanisms that lead to low-capacity traps without requiring explicit knowledge of the symmetry type or architecture.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.