Reproduce “Evaluating Agent-based Program Repair at Google” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Pat Rondon, Renyao Wei, José Cambronero, Jürgen Cito, Aaron Sun, Siddhant Sanyam, Michele Tufano, Satish Chandra

Researchers at Google conducted the first systematic study of an agent-based program repair system in a large industrial codebase, creating the GITS-Eval benchmark from internal bugs. A baseline agent, Passerine, successfully generated plausible fixes for 73% of machine-reported bugs and 25.6% of human-reported bugs, highlighting key differences between enterprise and open-source bugs.

Key insights

  • Enterprise bugs differ from open-source benchmarks (like SWE-Bench) by being more multilingual, having larger and more dispersed code changes, and containing fewer code terms in their descriptions, making localization harder.
  • Richer bug reports, particularly those with explicit reproduction information, significantly improve an agent's ability to localize faults and generate plausible patches.
  • Analysis of agent 'trajectory smells' (e.g., lack of test execution, repeated searches) provides actionable insights for designing more effective and efficient agent behaviors in future systems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.