Reproduce “Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shuyang Xie, Shuxiao Xie, Feng Zhu, Yanli Ji, Wangmeng Zuo
A framework positions off-the-shelf coding agents as test-suite auditors, identifying latent bugs in officially accepted solutions by generating adversarial test cases. This methodology verified 589 accepted-but-buggy submissions in AtCoder, and agent-generated test suites achieved 0.952 bug coverage on fresh Codeforces problems, outperforming various baselines.
Key insights
- Off-the-shelf coding agents can effectively serve as 'test-suite auditors,' generating adversarial tests capable of uncovering latent bugs in officially accepted code.
- A robust, independent certification chain, combining consensus from multiple reference solutions and strict input legality validation, is essential for reliably certifying bugs and preventing false positives from invalid test inputs.
- Agentic workflows enhance LLM performance in adversarial test generation, yielding sustained high marginal coverage in test sets compared to bare single-call models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.