Reproduce “Automated Discovery Has No Universally Superior Harness” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Akshat Gupta, Jermaine Lei, Alexander Lu, Gopala Anumanchipalli, Leshem Choshen
The authors compare 30 search harnesses for language-model-guided mathematical discovery, testing four models on three tasks with matched rollout budgets. No fixed harness reliably wins across the 12 model–problem pairs. They also evaluate pruning weaker runs using intermediate scores: under five full-run equivalents, the best schedule averages 85.75%, versus 84.54% for an unpruned portfolio, in empirical resampling.
Key insights
- No single fixed discovery harness universally outperforms others; optimal choices are model- and problem-dependent hyperparameters.
- Early discovery progress reliably predicts final performance, with Spearman rank correlations exceeding 0.70 at the 50% checkpoint for most problem pairs.
- Online adaptive allocation strategies, which dynamically prune less promising runs, consistently surpass static and non-adaptive ensemble approaches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.