Reproduce “VulScribeR: Exploring RAG-based Vulnerability Augmentation with LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Seyed Shayan Daneshvar, Yu Nong, Xu Yang, Shaowei Wang, Haipeng Cai

VulScribeR, a framework developed by researchers from the University of Manitoba, Washington State University, and the University at Buffalo, leverages Large Language Models with Retrieval-Augmented Generation (RAG) to generate diverse vulnerable code samples for Deep Learning-based Vulnerability Detection (DLVD) models. Its Injection and Extension strategies significantly boost DLVD performance, showing an average 30.80% F1-score improvement over no augmentation and achieving up to a 53.84% F1-score gain at scale with a cost of US$1.88 per 1,000 samples.

Key insights

  • Enriching the contextual environment of vulnerabilities through Injection and Extension strategies is more effective for DLVD models than simply mutating existing vulnerable code.
  • The Retrieval-Augmented Generation (RAG) component, especially the clustering of vulnerable samples, plays a crucial role in enhancing the quality and relevance of generated samples, improving F1-scores by up to 10.77% for Extension.
  • VulScribeR's augmentation method is highly scalable and cost-effective, consistently improving DLVD performance with increased generated samples, unlike previous baselines that degrade or plateau.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.