Reproduce “Characterizing the Failure Modes of LLMs in Resolving Real-World GitHub Issues” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yanjie Jiang, Yian Huang, Guancheng Wang, Junjie Chen, Hui Liu, Lionel Briand
This research systematically characterizes the failure modes of Large Language Models (LLMs) when resolving real-world GitHub issues, developing a five-stage taxonomy and identifying that models primarily struggle with synthesizing comprehensive repair logic (37%) and accurately interpreting problem specifications (29.2%). The study also highlights that 16.5% of failures stem from rigid evaluation harnesses rather than fundamental coding errors.
Key insights
- LLM failures are predominantly attributed to higher-level cognitive tasks like synthesizing comprehensive repair logic and interpreting problem specifications, rather than lower-level coding or localization issues.
- A substantial portion of failures (16.5%) stems from rigid evaluation harnesses and strict output format expectations, where functionally correct solutions are rejected due to a 'specification-oracle gap'.
- Models exhibit 'reasoning inertia' during failures, characterized by a sharp increase in prompt token usage without achieving resolution, indicating unproductive debugging loops.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.