Reproduce “On the Reliability of LLM-Based Vulnerability Patching Benchmarks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dang K Le, Wenxuan Shi, Xinyu Xing

The authors examine how evaluation choices affect automated vulnerability-repair benchmarks through a taxonomy, controlled multi-language dataset and harness, and experiments. They compare crash-triggering proof-of-concept tests with tests added or revised by upstream developers. In one experiment, Claude Opus 4.7 passed the proof of concept in 90.2% of cases but developer tests in 60.7%; these rates describe this dataset, not general correctness.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.