Reproduce “TLAPS-Bench: Formally Proving Correctness of Real-World System Specifications with TLA+ Proof System” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Qian Cheng, Ruize Tang, Saad Mohammad Rafid Pial, Zhizhen Cathy Cai, Yiming Su, Jackson Clark, Munim Thahmid, Mahdiya Rahman Sukanya and 8 more

TLAPS-Bench tests whether AI agents can construct machine-checked proofs for concurrent-system models, including invariants. Agents iteratively use proof libraries, model checkers, and TLAPS, but only TLAPS-verified arguments count. At max effort, GPT-6.1 Sol passed 31 of 70 targets (44.3%) on pass@3, meaning at least one of three runs; these are formal-model results, not proofs of deployed systems.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.