Reproduce “The Geometry of Neural Nets' Parameter Spaces Under Reparametrization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Agustinus Kristiadi, Felix Dangel, Philipp Hennig
Researchers from the Vector Institute and University of Tübingen present a Riemannian geometric framework to resolve inconsistencies observed in deep neural networks when their parameters are reparametrized. Their approach demonstrates that by explicitly accounting for the metric, properties such as Hessian-based flatness, gradient descent trajectories, and probability density modes become invariant to reparametrization, providing a robust theoretical foundation for analysis.
Key insights
- Invariance under reparametrization is an inherent property of neural networks that can be achieved by explicitly representing and correctly transforming the underlying Riemannian metric.
- Any preconditioned gradient descent method, not exclusively Natural Gradient Descent, can achieve invariant optimization trajectories if its associated metric is managed geometrically.
- Inconsistent modes of probability densities can be resolved by defining them with respect to a Riemannian volume form, ensuring invariance under reparametrization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.