Reproduce “Incentivizing LLMs to Self-Verify Their Answers” on OpenResearch

About this paper

Fuxiang Zhang, Jiacheng Xu, Chaojie Wang, Ce Cui, Yang Liu, Bo An

Researchers from Nanyang Technological University and Skywork AI developed a self-verification framework that trains Large Language Models to critically assess their own generated solutions within a unified reinforcement learning process. This approach overcomes limitations of external verifiers, enabling enhanced performance and efficiency in mathematical reasoning tasks by achieving strong greedy-decoding scores and outperforming baselines in test-time scaling.

Key insights

• Training LLMs to self-verify their solutions directly addresses the distribution discrepancy issue, as the internal verifier is inherently aligned with the generator's output distribution.
• Integrating the verification task into the RL problem-solving process intrinsically enhances the model's underlying reasoning capabilities, leading to improved greedy-decoding performance.
• The use of a policy-aligned buffer and a dynamic verification reward function is crucial for stable training and incentivizing the model to accurately verify both correct and challenging incorrect solutions.

Summary by alphaXiv.