Reproduce “DuaShepherd: Integrating Stepwise Correctness and Potential Rewards for Mathematical Reasoning” on OpenResearch

About this paper

Yuanhao Wu, Juntong Song, Hanning Zhang, Tong Zhang, Cheng Niu

Researchers from NewsBreak and the University of Illinois Urbana-Champaign developed DuaShepherd, a framework that integrates stepwise correctness and potential rewards to enhance Large Language Model mathematical reasoning. The system utilizes an automated pseudo-labeling pipeline and a multi-task learning architecture, yielding improved performance on MATH500 (e.g., 0.526 accuracy with Mistral-7B) and ProcessBench (65.5% F1 score) benchmarks compared to single-signal reward models.

Key insights

• Stepwise correctness (identifying past errors) and potential rewards (forecasting future success) are complementary signals that, when integrated, provide a more holistic evaluation of reasoning quality than either signal alone.
• Multi-task training with distinct heads for correctness and potential enables the shared base model to learn more robust and generalized representations, surpassing the performance of individually trained or pre-compounded reward models.
• Automated pseudo-labeling on large-scale datasets offers a scalable and cost-effective method to generate high-quality fine-grained process supervision, mitigating the reliance on expensive human annotations.

Summary by alphaXiv.