Reproduce “Deep kernel processes” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Laurence Aitchison, Adam X. Yang, Sebastian W. Ober

The authors introduce the deep inverse Wishart process (DIWP), which learns representations by sampling pairwise similarity matrices and applying nonlinear kernels between samples. It avoids carrying feature vectors through hidden layers. On CIFAR-10, a three-layer fully connected DIWP reaches 50.5 ± 0.1% test accuracy, versus 47.4 ± 0.1% for NNGP in matched comparisons; these results do not establish an advantage over convolutional networks.

Key insights

  • Gram matrices are invariant to the permutation and rotational symmetries that complicate variational inference for features or weights in DGPs and BNNs, simplifying the posterior structure for more tractable inference.
  • A wide range of deep probabilistic models, including DGPs with isotropic kernels, finite BNNs, infinite BNNs, and infinite BNNs with bottlenecks, can be formally expressed as instances of DKPs.
  • The DIWP allows learning a layer-specific "width" parameter ( \\delta_l ) which adaptively controls the variability of kernel transformations, ranging from deterministic to flexible, influencing the model's capacity to learn representations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.