Reproduce “Dyve: Thinking Fast and Slow for Dynamic Process Verification” on OpenResearch

About this paper

Jianyuan Zhong, Zeju Li, Zhijian Xu, Xiangyu Wen, Qiang Xu

Researchers at The Chinese University of Hong Kong developed Dyve, an adaptive dynamic process verifier for LLMs that integrates rapid token-level checks with deliberate multi-token analysis, inspired by Kahneman's cognitive theory. This approach achieved state-of-the-art F1 scores on mathematical reasoning benchmarks like 68.5 on GSM8K and 58.3 on MATH, and improved proposer LLM accuracy to 95.5% in Best-of-N settings, leveraging a novel consensus-filtered data curation technique.

Key insights

• An adaptive verification strategy, blending 'fast' intuitive checks with 'slow' deliberate analysis, effectively balances verification efficiency and accuracy, avoiding the pitfalls of uniform, computationally expensive deep analysis.
• High-quality process supervision data, critical for training robust verifiers, can be effectively and economically curated from large, noisy datasets through a multi-stage consensus filtering and LLM-as-Judge pipeline.
• Explicitly training the verifier to identify and flag uncertain or erroneous steps for deeper 'System 2' scrutiny significantly enhances its ability to detect subtle reasoning errors and improves overall performance.

Summary by alphaXiv.