Reproduce “Measuring the Influence of Incorrect Code on Test Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dong Huang, Jie M. Zhang, Mark Harman, Mingzhe Du, Heming Cui

The authors measure how code correctness affects language-model-generated tests by comparing prompts that pair the same task description with correct or incorrect implementations. Across 11 models and three benchmarks, correct-code prompts yield higher test-level accuracy, line coverage, and bug detection. On selected real-world cases, task-level bug detection is 13.5% versus 7.1%; these results cover only 41 retained tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.