Reproduce “Improving Value-based Process Verifier via Low-Cost Variance Reduction” on OpenResearch

About this paper

Zetian Sun, Dongfang Li, Baotian Hu, Min Zhang

Researchers at Harbin Institute of Technology (Shenzhen) developed COMpound Monte Carlo Sampling (ComMCS), a low-cost technique to reduce high variance in training labels for value-based process verifiers in large language models. This method consistently improved reasoning accuracy on MATH-500 and GSM8K benchmarks by up to 2.8 percentage points in Best-of-32 sampling without increasing LLM inference costs.

Key insights

• Monte Carlo estimators for binary outcomes in LLM reasoning are unbiased but inherently suffer from high variance due to limited sampling, which is the primary bottleneck for value-based process verifiers.
• Information from subsequent reasoning steps can be leveraged to reduce the variance of current value estimates while maintaining unbiasedness, drawing a parallel to Bellman equations in reinforcement learning.
• Modeling the full value distribution with cross-entropy loss, when combined with the proposed variance reduction, provides a more robust and effective approach than traditional regression or binary classification for verifier training.

Summary by alphaXiv.