Reproduce “POSTCONDBENCH: Benchmarking Correctness and Completeness in Formal Postcondition Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gehao Zhang, Juan Zhai
The POSTCONDBENCH benchmark introduces a multilingual, repository-level dataset for evaluating formal postcondition generation by LLMs, revealing a substantial and persistent gap between the correctness and behavioral completeness of generated specifications. It leverages mutation testing to enable automated completeness assessment, demonstrating that even state-of-the-art models frequently produce correct but behaviorally incomplete postconditions.
Key insights
- There is a persistent and substantial gap between the correctness and behavioral completeness of LLM-generated formal postconditions, with less than half of correct outputs typically being complete for most models.
- The most frequent failure modes for incorrect postconditions involve specification language misuse (54%) and semantic overreach (34%), while incompleteness is primarily caused by under-specified return value behaviors (78%).
- Method characteristics significantly impact performance; standalone methods (without repository dependencies) and shorter methods yield higher correctness and completeness, while increased dependency complexity and method length lead to performance degradation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.