Reproduce “HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Petr Simecek, Elnaz Babayeva, Jiri Balhar, Michal Bida, Michal Buran, Vaclav Cadek, Luigino Camastra, Tomas Dulka and 18 more
HoF-Bench, a new benchmark comprising 95 real-world, AI-discovered CVEs, quantifies the rediscovery capabilities of non-frontier LLM-based scanners in full production repositories. The evaluation revealed that multiple passes and model diversity substantially improve the detection of these vulnerabilities, with C-language code presenting the greatest challenge.
Key insights
- Vulnerability detection by LLMs is stochastic; repeated scanning (e.g., four passes) substantially increases cumulative CVE rediscovery, adding 12-20 CVEs per model on average.
- Leveraging a diverse set of detector models improves overall rediscovery, as different models exhibit distinct blind spots, covering 84 out of 95 CVEs collectively.
- Providing LLM-generated security context yielded only a modest 2.2 percentage point increase in rediscovery, but concurrently increased the volume of candidate findings by 25%.
- Programming language heavily influences detection difficulty; C-based CVEs are notably harder to rediscover (16–62% recall) compared to JavaScript (71–100%) and PHP (50–73%).
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.