Reproduce “Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Daniel Kunin, Javier Sagastuy-Brena, Surya Ganguli, Daniel L.K. Yamins, Hidenori Tanaka
The authors derive dynamics for parameter combinations in image classifiers by grouping weights according to translation, scale, and rescale symmetries rather than modeling weights. They combine gradient-flow conservation laws with finite-step corrections, weight decay, and minibatch gradient integrals. For VGG-16 on Tiny ImageNet, predictions closely track measured traces, but the analysis is not exact for discrete SGD trajectories or architectures.
Key insights
- Intrinsic architectural symmetries impose fundamental geometric constraints on neural network loss landscapes, unifying various prior observations on gradient and Hessian properties.
- These symmetries define specific, 'meaningful' combinations of parameters whose dynamics, even under practical SGD, are analytically tractable.
- Practical aspects of SGD (weight decay, finite learning rate, momentum) systematically perturb and 'break' ideal conservation laws in predictable ways, leading to solvable ODEs.
- Momentum can induce harmonic oscillations in parameter combinations, which can be modeled as driven harmonic oscillators, depending on hyperparameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.