Reproduce “LEVER: Learning to Verify Language-to-Code Generation with Execution” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, Xi Victoria Lin
LEVER is a framework that improves the accuracy of language-to-code generation by training a verifier model to evaluate candidate programs using their execution results. The system consistently enhanced the performance of various large language models across four diverse benchmarks, establishing new state-of-the-art results on all tested datasets.
Key insights
- While large language models can generate many code candidates, identifying the correct one from the set is critical, as a correct program is often present in the sampled outputs.
- Incorporating semantic features from program execution, such as data types and value ranges, provides valuable signals for verifying correctness beyond simple pass/fail checks.
- Training a smaller, specialized verifier model is significantly more data-efficient and computationally cost-effective than finetuning large generative code models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.