Reproduce “Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zerui Yang, Quan Xue, Miaomiao Dong, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu and 4 more
The authors replay recorded coding-agent runs, recover intermediate patches, score them with hidden tests, and compare stopping rules using inference-token costs. This evaluates rules on the same trajectories without changing agent behavior. In a preregistered replication of 386 Qwen3.6-27B mini-SWE-agent runs on SWE-bench Verified, twice-passed stopping saved 22.8% cost with a −1.0-point resolution change, but non-inferiority was not established.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.