Reproduce “Evolving Self-tuning Benchmarks for Long-Horizon Tool Use” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hyunji Alex Nam, Michael Dennis, Natasha Jaques, Minsuk Chang
TURTLE generates solver-relative tool-use tasks by mutating existing tasks and tools, then using solver attempts and feedback to guide further proposals. In AppWorld, Gemini 3.6 Flash rated the hard-task share for Gemma 4 31B-it’s set as rising from 19.8% to 52.5%; 99.9% of evolved tasks had 40–60% completion rates. The difficulty ratings are judge-assigned, and results are not model-independent.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.