Reproduce “Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamics” on OpenResearch

About this paper

Yehonatan Avidan, Qianyi Li, Haim Sompolinsky

Avidan, Li, and Sompolinsky developed an exact analytical theory for the learning dynamics of wide, deep neural networks, unifying the Neural Tangent Kernel (NTK) and Neural Network Gaussian Process (NNGP) frameworks. Their work introduced the Neural Dynamical Kernel (NDK), a time-dependent generalization that explains both gradient-driven and slow diffusive learning phases, offering insights into how network hyperparameters and activation functions influence performance and providing a theoretical basis for representational drift observed in biological brains.

Key insights

• The Neural Dynamical Kernel (NDK) serves as a unifying element, where the NTK corresponds to its instantaneous value and the NNGP kernel emerges from its long-time integral, establishing a fluctuation-dissipation like connection.
• Learning in wide networks unfolds in two distinct phases: an initial, fast gradient-driven stage dominated by loss minimization (NTK-like) and a subsequent slow, diffusive exploration stage (NNGP-like) that approaches Bayesian equilibrium.
• Hyperparameters like initialization variance (σ₀²) and prior variance (σ²) critically influence the learning trajectory and generalization, particularly for non-homogeneous activation functions like sigmoidal neurons, where they can cause significant performance shifts (e.g., up to 12% accuracy observed).

Summary by alphaXiv.