Reproduce “More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alexander Wei, Wei Hu, Jacob Steinhardt

This paper leverages random matrix theory to analyze how neural representations generalize, demonstrating that Generalized Cross-Validation accurately predicts generalization risk in realistic deep learning settings. It identifies that the alignment between kernel eigenvectors and the target function, rather than eigenvalue decay, is crucial for the generalization benefits observed in pretrained models.

Key insights

  • The Generalized Cross-Validation (GCV) estimator accurately predicts generalization performance in realistic neural network settings, providing precise and reliable estimates.
  • Pretrained neural networks generalize better than randomly initialized ones primarily due to better alignment between the eigenvectors of the kernel matrix and the target function, rather than faster eigenvalue decay.
  • Random matrix theory provides a theoretical foundation for GCV's effectiveness and explains empirical scaling laws in deep learning by relating generalization error to eigenvalue decay rate and alignment coefficients.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.