Reproduce “Hidden symmetries of ReLU networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
J. Elisenda Grigsby, Kathryn Lindsey, David Rolnick
The authors characterize when fully connected ReLU networks have settings identifiable up to neuron permutations and positive rescalings. They prove these settings form a positive-measure subset when every hidden layer is at least as wide as the input. At initialization, maximum functional dimension appears in 25% of depth-4 samples at joint width/input 5 and 66% at 15; these are not training results.
Key insights
- A significant class of ReLU network architectures (where hidden layers are not narrower than the input) can possess parameter settings where no hidden symmetries exist.
- Hidden symmetries stem from identifiable geometric and functional conditions within the network, such as unactivated neurons or the image of a layer being confined to a lower-dimensional subspace.
- The functional dimension effectively quantifies the degree of parameter redundancy, with a lower functional dimension indicating the presence of hidden symmetries beyond permutation and scaling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.