Reproduce “Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaojing Sun, Yuhan Zeng, Zihua She, Xiao Wang
The authors introduce REUSE, a method for iterative improvement that limits benchmark feedback to promotion decisions. It allocates a shared error budget across possible decision histories, under assumptions that the evaluation set is independent and provides no other feedback. In Covertype classification, REUSE had no held-out-measured false promotions and nearly matched empirical best-of-K’s final gain in one experiment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.