Reproduce “PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara, Martin Vechev
PROBE tests whether coding agents can find repository-wide bugs that maintainers later fix. It freezes repositories in time and lets agents inspect them without future history or network access. In the full-benchmark evaluation, GPT-5.6 Sol at high effort achieved 20.8% recall and 6.7% precision. A panel confirmed 84% of sampled unmatched findings as correct; scores use a partial reference set.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.