Reproduce “Stochastic Kernel Regularisation Improves Generalisation in Deep Kernel Machines” on OpenResearch

About this paper

Edward Milsom, Ben Anson, Laurence Aitchison

The authors present deep kernel machines that learn task-specific representations by learning Gram matrices of pairwise similarities rather than neural-network weights or features. They add stochastic kernel regularisation, which samples noisy Gram matrices during training. On CIFAR-10, the method reaches 94.52% test accuracy, matching an Adam-trained network but trailing an SGD-trained network; the result is limited to this benchmark.

Key insights

• Representation learning is a critical factor for achieving high performance in deep learning, as kernel-only methods capable of learning representations (DKMs) can match Adam-trained neural networks on complex tasks.
• Introducing controlled stochasticity into the learning process of inherently deterministic Gram matrices, akin to dropout, significantly improves generalization and numerical stability in deep kernel machines.
• Specific algorithmic modifications, such as Taylor approximations for objective terms and careful handling of linear algebra, can overcome the numerical instability of kernel methods, enabling efficient training with lower-precision hardware.

Summary by alphaXiv.