Reproduce “Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ryo Karakida, Shotaro Akaho, Shun-ichi Amari
Researchers from AIST and RIKEN Center for Brain Science applied a mean field approach to derive universal statistics for the Fisher Information Matrix (FIM) in deep neural networks, revealing that most FIM eigenvalues are nearly zero while the maximum eigenvalue scales linearly with network width, providing a quantitative basis for setting optimal learning rates.
Key insights
- The FIM eigenvalue distribution is highly skewed, with the majority of eigenvalues scaling as O(1/M) (approaching zero) but a few large eigenvalues, with the maximum scaling as O(M) with network width M.
- The parameter space of DNNs is locally flat in most dimensions but highly distorted in specific directions, which explains the complex topography of the loss landscape.
- The critical learning rate for stable convergence of gradient descent is inversely proportional to the maximum FIM eigenvalue, implying that wider and deeper networks require proportionally smaller learning rates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.