Reproduce “Fast and Scalable Spike and Slab Variable Selection in High-Dimensional Gaussian Processes” on OpenResearch

About this paper

Hugh Dance, Brooks Paige

SSVGP approximates spike-and-slab Gaussian process inference variationally, using inclusion probabilities, pruning, and minibatching to reduce computation. On a synthetic regression task with 1,000 inputs and six signals, it achieved median normalized prediction error 0.0068 and variable-selection MCC 1.0 across 50 trials. The MCMC reference matched these scores in one trial, with estimated runtime exceeding 10,000 seconds.

Key insights

• Approximating the Dirac spike with a narrow Gaussian distribution allows for closed-form coordinate ascent updates for variable inclusion probabilities in variational inference, making spike and slab GPs computationally feasible.
• Bayesian Model Averaging, weighted by LOOPD, offers a robust mechanism for adaptive sparsity and more reliable variable selection, mitigating the sensitivity of sparsity to the spike precision hyperparameter.
• The combination of variational inference with minibatching, zero-temperature approximations, and dropout pruning provides significant computational speed-ups, enabling spike and slab GPs to scale to previously intractable high-dimensional and large-scale problems.

Summary by alphaXiv.