Reproduce “Investigating Test Overfitting on SWE-bench” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, Martin Hirzel
Large Language Models generating code patches for repository-level issues often overfit to auto-generated tests, passing them while failing true golden tests. This overfitting is often exacerbated by iterative test-based refinement, where, for instance, GPT-4o exhibited a 33.0% initial overfitting rate, rising to 35.9% with refinement.
Key insights
- LLM-generated code patches consistently overfit to auto-generated tests, passing them but failing true acceptance criteria from hidden golden tests.
- Iterative code refinement based on these observed tests often exacerbates the overfitting problem, rather than mitigating it.
- Models exhibit a preference for modifying code over tests during refinement, and overfitted patches tend to have lower execution coverage, providing a potential indicator of overfitting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.