Reproduce “Change And Cover: Last-Mile, Pull Request-Based Regression Test Augmentation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zitong Zhou, Matteo Paltenghi, Miryung Kim, Michael Pradel
ChaCo introduces an adaptive, LLM-based method for generating regression tests specifically for uncovered code in pull requests, closing the "last-mile" test gap. It achieved 100% patch coverage for 30% of evaluated PRs in projects like SciPy, leading to high developer acceptance with 8 out of 12 submitted tests merged and the discovery of two previously unknown bugs, all at an average cost of $0.11 per PR.
Key insights
- Dynamically identifying and utilizing existing test context (e.g., relevant test methods and fixtures) is crucial, doubling coverage increment compared to LLM-only context.
- An iterative feedback loop, providing runtime results to the LLM, dramatically improves test quality, leading to a 312% higher pass rate and 460% higher coverage increment.
- LLMs can generate highly relevant and acceptable regression tests for specific code changes when provided with rich contextual information and guided by an iterative refinement process.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.