Reproduce “In-Context Decision Making for Optimizing Complex AutoML Pipelines” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Amir Rezaei Balef, Katharina Eggensperger

CASH+ extends automated machine learning beyond a single joint search space by allocating resources among heterogeneous, self-optimizing workflows. Its PS-PFN strategy uses prior-data fitted Transformers to predict each workflow’s future best performance, generally outperforming conventional max-bandit and Thompson-sampling baselines at low and intermediate budgets, while cost-aware variants complete 2.94%–17.35% more evaluations without consistently improving final performance.

Key insights

  • Sampling the posterior of a workflow’s future maximum is better aligned with AutoML’s best-found-performance objective than methods designed for immediate or cumulative reward.
  • PFNs can represent left-skewed, nonstationary, and arm-specific optimization trajectories more flexibly than the fixed distributional assumptions used by PS-Max.
  • Prior selection matters: mismatched trajectory priors can lead to excessive exploration or poor extrapolation, and greater evaluation throughput does not necessarily translate into better final solutions when cheap workflows are weaker.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.