Reproduce “A Fast, Well-Founded Approximation to the Empirical Neural Tangent Kernel” on OpenResearch

About this paper

Mohamad Amin Mohamadi, Wonho Bae, Danica J. Sutherland

The authors introduce the pseudo-neural tangent kernel (pNTK), which approximates a classifier’s empirical neural tangent kernel by computing the gradient kernel of the sum of its logits and repeating it across output classes. At initialization, a theorem bounds relative error at O(n^-1/2) with high probability under stated fully connected conditions. Experiments report lower resource costs on CIFAR-10, while trained-network support is empirical.

Key insights

• The pNTK offers a theoretically grounded alternative to heuristic "sum of logits" approximations, valid for finite-width networks at initialization.
• Kernel regression predictions made with pNTK rapidly converge to those from eNTK during training, even as underlying kernel matrix metrics between them may diverge, suggesting robustness of predictions.
• PNTK-based kernel regression frequently achieves higher test accuracy than full eNTK-based regression, indicating a potential regularization effect of the approximation.
• The dynamics of pNTK reveal that network representations can continue to evolve in later training stages for certain architectures (ConvNets, WideResNets), even after training error is minimized.

Summary by alphaXiv.