Reproduce “YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, Bernd Bischl

Researchers from LMU Munich introduced YAHPO Gym, a surrogate-based benchmark library for hyperparameter optimization (HPO) featuring over 700 multi-fidelity and multi-objective problems. This library provides approximately a 300x speedup for HPO method evaluation compared to real-world experiments while maintaining high fidelity to actual performance.

Key insights

  • Tabular benchmarks can produce unfaithful rankings of HPO methods due to the discretization of the search space, whereas surrogate benchmarks demonstrate closer alignment with real-world performance.
  • Multi-fidelity HPO methods generally improve performance in the intermediate stages of optimization, leading to faster progress compared to full-fidelity approaches.
  • In multi-objective HPO, simpler methods like Random Search with a quadrupled budget can be highly competitive in early optimization phases, though advanced methods like MEGO, ParEGO, and MIES achieve similar final performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.