Reproduce “FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Vikram Nitin, Baishakhi Ray, Roshanak Zilouchian Moghaddam

FaultLine, an LLM agent workflow from Columbia University and Microsoft, automates the generation of Proof-of-Vulnerability (PoV) tests by systematically reasoning about program data and control flow. The system achieved a 77% relative improvement over a state-of-the-art LLM agent baseline in generating correct PoV tests across a multi-lingual benchmark of 100 vulnerabilities.

Key insights

  • Structured, multi-step reasoning workflows, inspired by traditional program analysis, significantly improve LLM agent capabilities for tasks requiring deep code understanding like PoV generation.
  • Explicitly guiding LLM agents to reason about data flow from source to sink and control flow through branch conditions is critical for crafting inputs that accurately trigger complex vulnerabilities.
  • A robust feedback-driven repair loop, incorporating automated checks and iterative refinement, is essential for generating reliable and effective PoV tests from initial LLM-generated attempts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.