Reproduce “Spec-Harness: Assessing and Guiding LLM-Synthesized Formal Specifications Beyond Verifiability” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Md Rakib Hossain Misu, Cristina V. Lopes, Iris Ma, Yuangang Li, Shuvendu Lahiri

Spec-Harness evaluates contracts by separately testing preconditions and postconditions for correctness and completeness, then provides scores as synthesis feedback. Among 14,934 verifier-accepted specifications, only 35.6% met the threshold for both postcondition measures. In a held-out agent study, Codex’s FormalBench-120 score rose from 0.25 to 0.41 with feedback. These test- and mutation-based measures do not prove complete specification quality.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.