Reproduce “In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Herilalaina Rakotoarison, Steven Adriaensen, Neeratyoy Mallik, Samir Garibov, Edward Bergman, Frank Hutter
The Machine Learning Lab at the University of Freiburg introduces In-Context Freeze-Thaw Bayesian Optimization (ifBO), which utilizes a Prior-data Fitted Network (FT-PFN) to perform Bayesian learning curve extrapolation. This approach achieves state-of-the-art hyperparameter optimization performance while accelerating surrogate predictions by 10-100 times compared to existing methods, enabled by FT-PFN's ability to operate without online retraining and its meta-training exclusively on synthetic data.
Key insights
- Prior-data Fitted Networks (PFNs) can effectively replace online-trained surrogate models in multi-fidelity Bayesian Optimization, leading to significant speedups and improved stability.
- Meta-training a PFN on a sophisticated synthetic data prior, specifically designed to mimic real learning curve dynamics, enables it to generalize effectively to real-world HPO tasks without direct exposure to real data.
- Randomizing acquisition function parameters (horizon and threshold) provides a robust and parameter-free strategy for balancing exploration and exploitation in freeze-thaw HPO, especially with accurate but potentially conservative uncertainty estimates from PFNs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.