Reproduce “Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tural Mehtiyev, Wesley Assunção
The authors analyze 9,374 trajectories from 19 agents on 500 SWE-bench Verified tasks to examine failures and work patterns. Their review finds that 10 of 12 never-solved tasks with patches of at most 10 changes involved intervention at the wrong architectural level. Within tasks, resolved runs averaged 44.0 steps versus 39.6 for failed runs; these findings do not establish causation.
Key insights
- Coding agents exhibit an 'architectural reasoning gap,' failing on seemingly simple tasks by correctly localizing bugs but intervening at the wrong architectural layer (e.g., patching symptoms instead of root causes).
- Trajectory length is an ambiguous signal; while failed agents take more steps on average, successful agents on contested tasks take 10.0% *more* steps, indicating successful agents may invest more in complex tasks.
- An agent's underlying LLM capability is the primary driver of both its success rate and behavioral strategy, with the framework's influence diminishing as LLM capabilities improve.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.