Reproduce “CommitShield: Tracking Vulnerability Introduction and Fix in Version Control Systems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhaonan Wu, Yanjie Zhao, Chen Wei, Zirui Wan, Yue Liu, Haoyu Wang

COMMITSHIELD integrates Large Language Models (LLMs) with static analysis to precisely track software vulnerabilities through their lifecycle in version control systems. The system improves vulnerability fix detection (VFD) and vulnerability introduction detection (VID) by constructing detailed commit descriptions and comprehensive patch contexts, achieving F1-scores of 0.88 for VFD and 0.78 for VID.

Key insights

  • Combining LLMs with deep static analysis provides a powerful synergy, allowing the LLM to interpret nuanced code changes and natural language descriptions more accurately than either approach alone.
  • A comprehensive understanding of vulnerability introduction requires analyzing both added and deleted code, addressing the limitations of prior work focused solely on deleted lines.
  • Enriching commit descriptions with external information (issues, pull requests) and providing full functional and inter-procedural code context significantly boosts LLM performance for complex security tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.