Reproduce “Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Blake Bordelon, Abdulkadir Canatar, Cengiz Pehlevan
Analytical expressions for the generalization performance of kernel regression and wide neural networks are derived, decomposing the total error into contributions from individual spectral components. The work identifies a "spectral principle" where models sequentially learn successively higher spectral modes of target functions, predicting distinct learning stages and explaining the power-law scaling of generalization error with dataset size.
Key insights
- A "spectral principle" governs learning: as training data increases, kernel machines and wide neural networks progressively fit successively higher spectral modes of the target function, with lower-frequency modes learned faster.
- In high-dimensional settings, distinct learning stages emerge where specific frequency modes of the target function are learned in sequence, based on the scaling of training set size relative to input dimension.
- The power-law scaling of total generalization error with training samples is determined by the interplay between the target function's power spectrum and the kernel's eigenspectrum, consistent with observed empirical trends.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.