Reproduce “Parameter Symmetries Determine Representational Geometry in Overparameterized Nonlinear Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Marvin Theiss, Lukas Braun, Andrew M. Saxe, Erin Grant

The authors analyze how function-preserving parameter changes affect hidden representations in fully connected, one-hidden-layer nonlinear networks. They show that adding, duplicating, or rescaling features can change representational similarity matrices without changing the computed function. In a two-hidden-unit ReLU XOR example, same-function networks have nearly uncorrelated representations; this result concerns specified symmetries, not all equivalent networks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.