Reproduce “VulnGym: Benchmarking Coding Agents for Repository-Level Vulnerability Detection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kexing Ji, Jiachen Liu, Enze Hu, Cuiyun Gao, Keke Lian, Yongheng Liu, Lei Zhang, Tian Dong and 2 more
VulnGym introduces a new benchmark for evaluating coding agents on repository-level vulnerability detection, featuring fine-grained, line-level annotations of entry points, critical operations, and execution traces. Empirical evaluation reveals that current agents struggle with autonomous repository exploration, precise localization, and accurate trace construction, especially for complex vulnerabilities.
Key insights
- Current coding agents exhibit significant limitations in end-to-end, repository-level vulnerability detection, struggling particularly with autonomous code exploration and precise localization in complex scenarios.
- Diagnostic analysis reveals that identifying entry points (EPs) is generally more challenging for agents than localizing critical operations (COs), and trace construction often yields incomplete and noisy results.
- Agent performance for repository-level tasks is highly dependent on the specific combination of the underlying Large Language Model (LLM) and the agentic scaffold used.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.