Reproduce “Executable Prefix Values for Locating Agent Failures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bingyu Shen, Jiaqi Tang, Lan Wei, Yi Sheng, J. Jenny Li, Boyang Li
The authors measure whether a coding agent’s chance of solving a task falls at judge-labeled steps by restarting it from reconstructed histories and testing fresh continuations. Of 95 labeled steps probed, 64 showed no drop and four showed a drop of at least 0.2. Offline, value-based restarts raised expected success from 0.44 to 0.57 on selected tasks, under study assumptions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.