Reproduce “DAG-Math: Graph-of-Thought Guided Mathematical Reasoning in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuanhe Zhang, Ilja Kuzborskij, Jason D. Lee, Chenlei Leng, Fanghui Liu
Researchers at the University of Warwick and collaborating institutions introduced DAG-Math, a framework formalizing Large Language Model (LLM) mathematical reasoning as a rule-based stochastic process on Directed Acyclic Graphs (DAGs). This work developed novel metrics, including Perfect Reasoning Rate (PRR), to evaluate step-level logical coherence and revealed a statistically significant gap between final answer accuracy and the logical integrity of intermediate reasoning steps across various LLMs.
Key insights
- LLMs often achieve correct answers through exploratory search or by including irrelevant steps, as evidenced by a consistent and statistically significant gap between final answer accuracy (PASS@1) and Perfect Reasoning Rate (PRR).
- The structural characteristics of generated DAGs (e.g., size, density, branching) serve as diagnostic indicators of reasoning quality, with perfect reasoning trajectories yielding smaller, denser DAGs and incorrect ones showing larger, sparser, and highly branched structures.
- Each LLM exhibits a 'difficulty boundary' beyond which its reasoning process tends to 'explode' into highly branched, sparse DAGs, indicating a loss of coherent reasoning capacity for more challenging problems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.