Reproduce “Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification” on OpenResearch

About this paper

Zhenwen Liang, Ye Liu, Tong Niu, Xiangliang Zhang, Yingbo Zhou, Semih Yavuz

Researchers from the University of Notre Dame and Salesforce AI developed Math-Rev and Code-Rev, verifiers trained on diverse correct and incorrect solutions, to enhance Large Language Model reasoning. Their novel CoTnPoT collaborative verification strategy, which integrates Chain-of-Thought with Program-of-Thought, enabled open-source LLMs to achieve 95.6% on GSM8k and 76.9% on the MATH dataset, surpassing GPT-4o's performance on MATH.

Key insights

• Reference-free preference tuning methods, particularly SimPO, are more effective for training verifiers than outcome reward models or DPO, as they optimize directly for distinguishing correct from incorrect solutions without the overhead of reference models.
• Verifiers trained on diverse datasets of both correct and incorrect solutions demonstrate strong generalizability across different LLM reasoners, including those not seen during training.
• The collaborative CoTnPoT strategy, which combines interpretive (CoT) and executable (PoT) reasoning, significantly enhances error detection, especially by filtering solutions where CoT's final answer doesn't match PoT's execution.

Summary by alphaXiv.