Reproduce “Invariance Properties of the Natural Gradient in Overparametrised Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jesse van Oostrum, Johannes Müller, Nihat Ay
This paper analyzes how the natural gradient's invariance properties behave in overparametrised systems, which are prevalent in deep learning. It clarifies when the natural parameter gradient accurately reflects the true natural gradient on the model space and distinguishes between parametrisation independence and reparametrisation invariance in these complex settings.
Key insights
- The natural parameter gradient is not parametrisation independent, implying different parametrisations can lead to different natural parameter gradients.
- The natural parameter gradient is reparametrisation invariant from the model perspective, meaning its pushforward is independent of the choice of parametrization related by a diffeomorphism.
- The equality between the natural parameter gradient and the natural gradient on the model space holds under conditions where the parametrization is proper and the point is non-singular.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.