Reproduce “Why neural networks find simple solutions: the many regularizers of geometric complexity” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Benoit Dherin, Michael Munn, Mihaela Rosca, David G.T. Barrett

The authors introduce geometric complexity (GC), the dataset-average squared input-to-logit Jacobian, to measure how sensitively a network’s logits change on measured examples. They connect finite-step gradient descent to conditional pressure on this input-space variation and examine training choices and model width. In their experiments, GC tracks training patterns, including double descent, but does not establish a cause of generalization.

Key insights

  • GC acts as a unifying metric, showing that a wide range of explicit and implicit regularization techniques consistently reduce the geometric complexity of learned functions.
  • Deep networks, especially deeper ones, initialize to functions very close to zero with low GC, and training processes further drive GC downwards, suggesting an inherent bias towards geometrically simple solutions.
  • The discrete nature of gradient descent introduces an implicit gradient regularizer (IGR) whose strength, influenced by learning rate and batch size, transfers regularizing pressure to GC, leading to lower GC and better generalization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.