Reproduce “On the Reliability of LLM-Based Vulnerability Patching Benchmarks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dang K Le, Wenxuan Shi, Xinyu Xing
The authors examine how evaluation choices affect automated vulnerability-repair benchmarks through a taxonomy, controlled multi-language dataset and harness, and experiments. They compare crash-triggering proof-of-concept tests with tests added or revised by upstream developers. In one experiment, Claude Opus 4.7 passed the proof of concept in 90.2% of cases but developer tests in 60.7%; these rates describe this dataset, not general correctness.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.