Reproduce “Spec-Harness: Assessing and Guiding LLM-Synthesized Formal Specifications Beyond Verifiability” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Md Rakib Hossain Misu, Cristina V. Lopes, Iris Ma, Yuangang Li, Shuvendu Lahiri
Spec-Harness evaluates contracts by separately testing preconditions and postconditions for correctness and completeness, then provides scores as synthesis feedback. Among 14,934 verifier-accepted specifications, only 35.6% met the threshold for both postcondition measures. In a held-out agent study, Codex’s FormalBench-120 score rose from 0.25 to 0.41 with feedback. These test- and mutation-based measures do not prove complete specification quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.