Reproduce “An Exact Kernel Equivalence for Finite Classification Models” on OpenResearch

About this paper

Brian Bell, Michael Geyer, David Glickenstein, Amanda Fernandez, Juston Moore

Researchers from Los Alamos National Laboratory and collaborating universities developed the Exact Path Kernel (EPK), which provides the first exact kernel representation for finite-size classification models trained with discrete gradient descent. This framework replicates model predictions with machine precision and reveals non-intuitive properties, such as high confidence in regions distant from training data, offering new insights into neural network generalization.

Key insights

• Any finite-size parametric classification model trained with discrete gradient descent steps can be exactly represented as a kernel machine, extending theoretical understanding beyond infinite-width approximations.
• Prior approximate path kernels and NTK introduce quantifiable errors compared to the EPK, demonstrating that their idealizations significantly deviate from real-world training dynamics.
• The kernels implicitly learned by finite neural networks exhibit non-stationary properties and can predict with high confidence in regions far from training data, offering insights into generalization and robustness issues.

Summary by alphaXiv.