Reproduce “Agentic Software Issue Resolution with Large Language Models: A Survey” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhonghao Jiang, David Lo, Zhongxin Liu
A systematic survey of 242 studies maps the emerging field of large-language-model agents for software issue resolution across benchmarks, system designs, training methods, and empirical evaluation. It organizes the task into repository preprocessing, issue localization, patch generation, validation, and patch selection, while highlighting limitations of test-based success rates, benchmark contamination, and the growing use of reinforcement learning.
Key insights
- Benchmark development is expanding from Python bug fixing toward multilingual, multimodal, feature, performance, refactoring, mobile, and sequential-change tasks, but realism, reproducibility, and contamination remain competing objectives.
- Execution-based metrics dominate the literature: Resolved% or Pass@k appears in 76.4% of surveyed studies, yet a test-passing patch can still be behaviorally incorrect or difficult to maintain.
- Training is shifting toward executable-environment reinforcement learning; 24 of 39 representative models released after February 2025 use reinforcement learning, although reward quality is constrained by the reliability of tests and intermediate correctness signals.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.