Reproduce “LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning” on OpenResearch

About this paper

Yuqiang Sun, Daoyuan Wu, Yue Xue, Han Liu, Wei Ma, Lyuye Zhang, Yang Liu, Yingjiu Li

Researchers at multiple universities and MetaTrust Labs introduced LLM4Vuln, a framework and benchmark (UniVul) for disaggregating and improving Large Language Models' (LLMs) vulnerability detection abilities. Their work quantifies how external knowledge, code context, and prompt engineering influence different LLMs' reasoning, ultimately leading to the identification of 14 zero-day vulnerabilities in a real-world pilot study.

Key insights

• The impact of knowledge enhancement varies heterogeneously: it significantly boosts traditional foundation models for logic-based vulnerabilities (Solidity) but provides limited or negative gains for deep reasoning models and traditional language vulnerabilities.
• Context supplementation offers inconsistent improvements, often providing only slight gains or even degrading performance for deep reasoning models, suggesting that irrelevant context can be distracting.
• Chain-of-Thought (CoT) prompting reliably improves precision and reduces false positives across most models and languages, making LLM vulnerability predictions more dependable.

Summary by alphaXiv.