Reproduce “The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
James B. Simon, Madeline Dickens, Dhruva Karkada, Michael R. DeWeese
Researchers developed the Eigenlearning Framework, a theoretical approach to Kernel Ridge Regression (KRR) and wide neural networks (NNs) that simplifies derivations of generalization metrics by introducing a "conservation law" for "learnability". This framework offers new interpretations of KRR's behavior and explains phenomena such as the deep bootstrap, while also providing tools for analyzing adversarial robustness.
Key insights
- The total "learnability," summed over any complete basis of target functions, is conserved and equals the number of training examples $n$ at zero ridge parameter, or is bounded by $n$ at a positive ridge.
- KRR operates as an explicit competition among kernel eigenmodes for a fixed budget of learnability, where modes with high eigenvalues are well-learned and new samples primarily benefit partially-learned modes.
- A tight analogy exists between KRR generalization and the free Fermi gas in statistical physics, revealing universal sigmoidal learning curves for eigenmode learnability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.