Reproduce “Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel” on OpenResearch

About this paper

Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, Surya Ganguli

An empirical study from Stanford University and Element AI characterized the deep learning process as a universal "chaotic to stable" transition within the initial few epochs, where the final loss basin is determined and the Neural Tangent Kernel (NTK) rapidly acquires data-dependent features. This early phase, despite rapid kernel learning, still shows a performance advantage for full nonlinear training.

Key insights

• Deep neural network training universally undergoes an early "chaotic to stable" transient within the first few epochs.
• During this chaotic phase, the final low-loss basin is determined, and the Neural Tangent Kernel (NTK) undergoes its most rapid and impactful feature learning.
• A persistent "nonlinear advantage" for full training over learned NTK training, even at very low learning rates, indicates that practical deep learning dynamics are not fully captured by the NTK limit during early training.

Summary by alphaXiv.