Reproduce “VerityMath: Advancing Mathematical Reasoning by Self-Verification Through Unit Consistency” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vernon Toh Yan Han, Ratish Puduppully, Nancy F. Chen
VerityMath introduces a method that integrates unit consistency self-verification into program-aided language models (PAL) for mathematical problem-solving. It enables open-source LLMs to track and verify units in intermediate steps to improve robustness in handling multi-unit quantities, though current 7B models show a slight dip in overall accuracy compared to PAL baselines due to challenges in generating correct unit annotations.
Key insights
- Integrating a formal system of dimensional analysis as a self-verification mechanism within program generation offers a robust way to ensure physical and logical soundness in quantitative reasoning.
- Smaller 7B LLMs find the 'meta-task' of correctly generating unit annotations and `assert` statements challenging, often making errors in unit specification even when underlying arithmetic is correct.
- The performance of UCPs is highly sensitive to the size and quality of the training dataset, suggesting significant untapped potential with more extensive and accurate UCP annotations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.