Reproduce “Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yikai Wu, Xingyu Zhu, Chenwei Wu, Annie Wang, Rong Ge

This research from Duke University and Princeton University introduces a 'decoupling conjecture' to explain the empirically observed low-rank structure of the Hessian matrix in neural networks, proposing that layer-wise Hessians can be approximated by a Kronecker product of smaller, structured matrices. The work provides theoretical and empirical evidence that the top eigenspace of the Hessian is highly correlated across independently trained models and demonstrates its utility by deriving tighter PAC-Bayes generalization bounds.

Key insights

  • The input auto-correlation matrix (E[x^{(p)}x^{(p)T}]) for ReLU layers is approximately rank-1, with its top eigenvector close to the normalized mean input.
  • The output Hessian matrix (E[M^{(p)}_x]) is approximately rank (c-1), where 'c' is the number of classes, meaning only a few eigenvalues are significantly large.
  • The top eigenvectors of layer-wise Hessians, when reshaped into matrices, are approximately rank-1, and surprisingly, the top eigenspaces show high overlap across independently trained neural networks due to the consistent non-negative nature of layer inputs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.