Reproduce “PRIME: A Process-Outcome Alignment Benchmark for Verifiable Reasoning in Mathematics and Engineering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiangfeng Wang, Hangyu Guo, Yanlin Lai, Mitt Huang, Liang Zhao, Chengyuan Yao, Yinmin Zhang, Qi Han and 6 more
Researchers from StepFun, University of Science and Technology of China, and Tsinghua University introduce PRIME, a benchmark designed to evaluate Large Reasoning Model (LRM) verifiers by assessing both the correctness of the final answer and the logical soundness of the derivation process in STEM problems. This benchmark quantifies the prevalence of "lucky guesses" at 16.98% in LRM responses and demonstrates a strong correlation (R = 0.937) between verifier performance on PRIME and improved downstream LRM training effectiveness.
Key insights
- "Lucky guesses," where LRMs produce correct outcomes from flawed reasoning processes, are prevalent, accounting for 16.98% of generated responses, underscoring the limitations of outcome-centric verification.
- There is a strong positive linear correlation (R > 0.92) between a verifier's performance on the PRIME benchmark's overall accuracy (process-outcome consistency) and its effectiveness in improving downstream LRM performance during RLVR training.
- Incorporating process consistency into RLVR reward functions (process-aware RLVR) consistently and significantly improves LRM performance on STEM benchmarks compared to outcome-only baselines.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.