Reproduce “Identifying Helpful Context for LLM-based Vulnerability Repair: A Preliminary Study” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gábor Antal, Bence Bogenfürst, Rudolf Ferenc, Péter Hegedűs
This study from the University of Szeged investigates how different types of contextual information affect GPT-4o's performance in automated vulnerability repair using the Vul4J dataset. It demonstrates that providing raw CVE descriptions and manually extracted project-specific code context significantly improves repair rates, with an ensemble of top-performing prompts successfully fixing 61.9% of distinct vulnerabilities.
Key insights
- Providing relevant external context, especially raw CVE descriptions and manually extracted project-specific hints, significantly boosts LLM-based vulnerability repair rates.
- The length or verbosity of prompts has minimal impact on repair performance, and LLM-generated inline documentation can even be detrimental, suggesting that more context is not always better.
- An ensemble strategy, combining fixes from multiple distinct high-performing prompts, dramatically increases the overall number of unique vulnerabilities successfully repaired.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.