Reproduce “HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hossein Hajipour, Lea Schönherr, Thorsten Holz, Mario Fritz
HexaCoder, from CISPA Helmholtz Center for Information Security, introduces an oracle-guided data synthesis pipeline and a two-step code generation process to enable Large Language Models (LLMs) to produce more secure code. It successfully reduced generated security vulnerabilities by up to 85% on the CodeLMSec and Pearce et al. benchmarks while preserving functional correctness on HumanEval.
Key insights
- Oracle-guided synthesis effectively addresses data scarcity, generating 1,776 unique fixed code samples from 2,042 vulnerable ones.
- Providing detailed security reports (CodeQL reports + security hints) significantly increases the LLM's ability to repair vulnerabilities, achieving an 83.99% repair rate.
- A two-step generation process, allowing for initial security-relevant library suggestions, further reduces vulnerabilities by over 50% compared to a one-step approach on fine-tuned models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.