Reproduce “Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape” on OpenResearch

About this paper

Johanni Brea, Berfin Simsek, Bernd Illing, Wulfram Gerstner

This research from EPFL elucidated how neuron permutation symmetries in deep neural networks generate specific structures in the loss landscape, identifying critical points termed "permutation saddles" that reside within high-dimensional "equal-loss plateaus." The study demonstrated the existence of low-loss paths connecting equivalent global minima via these saddles and provided a combinatorial lower bound for their abundance, showing it increases with network size.

Key insights

• Permutation symmetries in deep networks are a direct source of an abundance of critical points, specifically "permutation saddles," which are embedded in high-dimensional equal-loss plateaus.
• The loss landscape's equivalent global minima are connected through paths of constant loss, implying an extensive network of "equal-loss valleys" that can be traversed by optimizers.
• The number of these symmetry-induced K-th order permutation points scales combinatorially with network size, providing a structural explanation for the large number of saddles and plateaus observed empirically.

Summary by alphaXiv.