Reproduce “Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zenan Li, Yifan Wu, Zhaoyu Li, Xinming Wei, Xian Zhang, Fan Yang, Xiaoxing Ma
Researchers from Nanjing University, Peking University, University of Toronto, and Microsoft Research Asia developed a framework that enhances autoformalization accuracy by large language models by selecting the most consistent formalization from multiple candidates. The method combines symbolic equivalence checks, leveraging automated theorem provers, with semantic consistency measured by re-informalization and embedding similarity, boosting pass@1 accuracy by up to 22.6% for Mistral-7B and 3.3% for GPT-4 on the MATH dataset.
Key insights
- Symbolic equivalence, relying on formal logical proofs, effectively identifies functionally identical formalizations despite superficial differences like variable names.
- Semantic consistency, by comparing re-informalized outputs with the original natural language, complements symbolic methods by catching subtle semantic drift or trivial formalizations.
- Combining symbolic and semantic consistency measures provides a more robust and comprehensive evaluation, leveraging their complementary strengths to significantly improve formalization selection.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.