Reproduce “Towards Learning Universal Hyperparameter Optimizers with Transformers” on OpenResearch

About this paper

Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Qiuyi Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski and 4 more

OptFormer represents diverse hyperparameter-tuning tasks and trial histories as sequences for one Transformer. It learns from heterogeneous offline records to suggest settings and predict responses. In HPO-B few-shot prediction tests, it outperforms a Gaussian Process in log-predictive likelihood and calibration error. For optimization, its strongest results require pairing the learned prior with Expected Improvement; the paper establishes no wall-clock speedup.

Key insights

• Transformer models can serve as a universal interface for modeling heterogeneous HPO experimental data, generalizing across vastly different search spaces.
• Combining a learned prior policy with model-based planning using acquisition functions provides superior optimization performance compared to pure behavioral cloning.
• Large-scale, diverse HPO data, even across varied domains, can be effectively leveraged by sequence models to learn robust HPO priors and policies.

Summary by alphaXiv.