Reproduce “Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhitong Xu, Haitao Wang, Jeff M Phillips, Shandian Zhe
This paper demonstrates that standard Gaussian Process-based Bayesian Optimization is effective for high-dimensional problems when using appropriate kernels or robust hyperparameter initialization. It identifies gradient vanishing during length-scale optimization as the primary reason for the Square Exponential kernel's poor performance in high dimensions and shows that the Matérn kernel or a simple robust initialization strategy can resolve this, achieving results comparable to or better than specialized high-dimensional BO methods.
Key insights
- Standard Bayesian Optimization (BO) with the Matérn-5/2 kernel consistently achieves strong performance in high dimensions, challenging the long-held belief that it inherently struggles.
- The perceived failure of standard BO with the Square Exponential (SE) kernel in high dimensions is primarily due to gradient vanishing during GP hyperparameter optimization, caused by improper length-scale initialization.
- The Matérn kernel's gradient structure makes it inherently more robust to high dimensions and less susceptible to gradient vanishing compared to the SE kernel.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.