Reproduce “TLAPS-Bench: Formally Proving Correctness of Real-World System Specifications with TLA+ Proof System” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qian Cheng, Ruize Tang, Saad Mohammad Rafid Pial, Zhizhen Cathy Cai, Yiming Su, Jackson Clark, Munim Thahmid, Mahdiya Rahman Sukanya and 8 more
TLAPS-Bench tests whether AI agents can construct machine-checked proofs for concurrent-system models, including invariants. Agents iteratively use proof libraries, model checkers, and TLAPS, but only TLAPS-verified arguments count. At max effort, GPT-6.1 Sol passed 31 of 70 targets (44.3%) on pass@3, meaning at least one of three runs; these are formal-model results, not proofs of deployed systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.