Reproduce “Limitations of the Empirical Fisher Approximation for Natural Gradient Descent” on OpenResearch

About this paper

Frederik Kunstner, Lukas Balles, Philipp Hennig

This paper critically analyzes the widely used empirical Fisher (EF) approximation in natural gradient descent, demonstrating its theoretical limitations as a second-order preconditioner. It shows that EF often fails to approximate the true Fisher Information Matrix or the Hessian, particularly near optima under model misspecification or far from optima, and suggests "variance adaptation" as an alternative explanation for its practical utility.

Key insights

• The empirical Fisher, unlike the true FIM or canonical GGN, often converges to zero at optima, failing to capture the non-zero curvature information of the Hessian.
• EF only approximates the true FIM at a minimum under stringent conditions (model realizability, sufficient data) that are rarely met in practical deep learning scenarios, especially with overparameterized and potentially misspecified models.
• The practical success of EF-based methods might stem from 'variance adaptation' to gradient noise in stochastic optimization rather than its effectiveness as a curvature-aware preconditioner.

Summary by alphaXiv.