Reproduce “Vul4Py: Benchmarking Automated Vulnerability Repair in Python with Paired Exploit and Functional Oracles” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tan Bui, Ting Zhang, Ferdian Thung, Yunpeng Xiong, Penghao Jiang, Xin Zhou, David Lo
Researchers from Singapore Management University, Monash University, and the University of New South Wales introduce Vul4Py, the first Python-specific benchmark for automated vulnerability repair (AVR) featuring mandatory paired exploit and functional oracles. Their evaluation demonstrates that software engineering agents achieve five to ten times higher repair success than directly prompted LLMs or specialized tools, while the paired oracle reliably filters 12.6% of patches that introduce functional regressions.
Key insights
- Software engineering agents that can iteratively interact with the project environment, execute tests, and refine patches are five to ten times more effective at repairing Python vulnerabilities than single-shot LLM prompts or specialized tools.
- Relying solely on exploit tests for evaluating AVR solutions is insufficient, as a significant portion (12.6% in this study) of seemingly successful patches introduce functional regressions by breaking unrelated program functionality.
- The proposed paired oracle methodology provides a robust and high-precision (94% accuracy) proxy for genuinely correct vulnerability repair, offering a more trustworthy evaluation standard for AVR benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.