Reproduce “Towards Automated Verification of LLM-Synthesized C Programs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Prasita Mukherjee, Benjamin Delaware

Purdue University researchers developed SYNVER, a framework that synthesizes C programs and machine-checked proofs of their correctness by combining large language models for code generation with formal verification using VST and separation logic. The system achieved full verification for 34 out of 48 C program benchmarks, including those with complex heap-manipulating data structures.

Key insights

  • Explicit syntactic biases in LLM prompts (e.g., 'no loops, use recursion') are crucial for generating programs amenable to automated formal verification with VST.
  • A hybrid proof generation strategy, combining domain-specific symbolic automation with LLM-guided tactic suggestions and backtracking, significantly enhances the automation of complex VST proofs.
  • LLMs are most effective in program generation when provided with a multi-modal input that includes natural language descriptions, structural biases, and formal specifications, rather than formal specifications alone.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.