Reproduce “RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hanyu Li, Yichi Zhang, Speed Zhu, Hang Su, Jun Zhu, Yinpeng Dong
The paper tests whether software agents can reason over repository context, not just fix visible issues. RepoMirage perturbs dependency paths, runtime targets, and local values, then creates explicit structural tasks while preserving tested functionality. Across eight models, original issue resolution averages 66.80%, versus 25.25% on Extend; this is not a same-task comparison, and results are limited to Python repository-modification tasks.
Key insights
- Code agents exhibit 'exploration drift' under increased context demands: they expand file exploration but struggle to convert gathered multi-file information into actionable structural understanding for effective problem-solving.
- The primary bottleneck in repository context reasoning is the failure to convert observed files into an actionable structural understanding, as evidenced by significant performance improvements when agents are provided with explicit structural hints.
- Strong end-to-end performance on standard benchmarks does not equate to robust repository context reasoning, indicating a critical gap in current evaluation methodologies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.