Reproduce “On the Power-Law Hessian Spectrums in Deep Learning” on OpenResearch

About this paper

Zeke Xie, Qian-Yuan Tang, Yunfeng Cai, Mingming Sun, Ping Li

This research from Baidu Research and RIKEN Center for Brain Science reveals that the Hessian spectrums of well-trained deep neural networks consistently follow a power-law distribution, an observation absent in randomly initialized networks. The paper provides a theoretical derivation of this power-law using a maximum entropy principle and demonstrates that the power-law slope correlates with model generalization. A striking finding is the spectral similarity between DNNs and protein vibrational modes, both exhibiting similar power-law exponents.

Key insights

• The power-law structure of the Hessian spectrum can be theoretically explained by maximizing a combined entropy measure, linking deep learning loss landscapes to statistical physics principles.
• The power-law decay of Hessian eigengaps clarifies why effective deep learning occurs in a robust, low-dimensional subspace, as larger eigengaps ensure stability of top principal components.
• The shared power-law spectral properties between deep neural networks and proteins suggest a potential universal mechanism governing complex adaptive systems in both artificial and biological domains.

Summary by alphaXiv.