Reproduce “OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xiaozhe Li, Jixuan Chen, Xinyu Fang, Shengyuan Ding, Haodong Duan, Qingwen Liu, Kai Chen

OPT-BENCH compares feedback use across 20 machine-learning tasks and 10 classical NP-hard problems. OPT-Agent drafts solutions, receives feedback, and uses history to improve valid solutions or debug failures. At 20 steps, GPT-4o with history beats its memoryless baseline on 18 of 20 machine-learning tasks, but wins only 4 of 10 NP task comparisons.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.