Reproduce “Deep Convolutional Networks as shallow Gaussian Processes” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Adrià Garriga-Alonso, Carl Edward Rasmussen, Laurence Aitchison

The authors derive a single Gaussian process whose recursively computed kernel represents a convolutional neural network in the infinite-filter limit, including residual connections. This preserves convolutional structure and permits exact Gaussian-process inference, though full kernel-matrix computation remains costly. On MNIST, their 32-layer ResNet-equivalent GP reports 0.84% test error, below a convolutional GP baseline, but not parametric neural-network methods.

Key insights

  • Deep convolutional networks and ResNets can be understood as Gaussian Processes in the infinite-width limit, providing a principled framework for exact Bayesian inference in these powerful architectures.
  • The translation-invariant structure of CNNs is preserved in their equivalent GP kernels, enabling efficient computation of the kernel matrix via convolutional operations, significantly improving scalability.
  • Empirical evidence shows that finite-width CNNs rapidly converge to their GP limit; networks with as few as 30-100 channels closely approximate the Gaussian output distributions and moments predicted by the infinite-width theory.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.