Reproduce “Just-in-Time Catching Test Generation at Meta” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Matthew Becker, Yifei Chen, Nicholas Cochran, Pouyan Ghasemi, Abhishek Gulati, Mark Harman, Zachary Haluza, Mehrdad Honarkhah and 7 more
The authors generate temporary tests for submitted code changes, targeting behavioral differences from parent revisions. The intent-aware workflow infers a change’s purpose and implementation risks, generates tests against risk-based mutants, and runs them on the proposed change. Among evaluated diffs, it found weak catches for 49/617 (7.9%), versus 26/654 (4.0%) with dodgy-diff generation; these are candidate differences, not verified bugs.
Key insights
- Diff-aware test generation, particularly the intent-aware approach, is significantly more effective at generating relevant test failures (weak catches) in new code changes than diff-unaware baseline methods.
- Combining LLM-based judgments with rule-based heuristics provides a robust and complementary approach to accurately assess test failures, drastically reducing the human effort needed to identify true bugs.
- Proactively catching bugs at the unit test level within the CI pipeline can prevent severe production failures with minimal developer friction, addressing a critical gap in traditional quality assurance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.