Reproduce “A Study of Bayesian Neural Network Surrogates for Bayesian Optimization” on OpenResearch

About this paper

Yucen Lily Li, Tim G. J. Rudner, Andrew Gordon Wilson

A study from New York University systematically evaluates Bayesian Neural Networks as surrogate models for Bayesian Optimization, demonstrating their competitiveness and superior performance over Gaussian Processes, particularly Infinite-Width BNNs, on challenging high-dimensional and non-stationary optimization tasks.

Key insights

• No single surrogate model consistently outperforms all others across diverse optimization problems; optimal performance often requires tailoring the model's inductive biases to the specific problem characteristics.
• Infinite-Width Bayesian Neural Networks (I-BNNs) prove particularly promising, especially for high-dimensional optimization, due to their ability to provide powerful non-Euclidean and non-stationary similarity metrics.
• While Hamiltonian Monte Carlo (HMC) provides high-quality inference for finite-width BNNs, full stochasticity may not always be necessary, with partially stochastic models like Deep Kernel Learning (DKL) remaining competitive.

Summary by alphaXiv.