Reproduce “Process Reward Models That Think” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, Lu Wang
ThinkPRM trains a reasoning model to verify each solution step using synthetic critiques filtered against gold process labels, yielding 1,000 chains and 8,000 labels. Unlike outcome-based filtering, it checks every step judgment. On ProcessBench’s OlympiadBench subset, ThinkPRM-14B achieves 87.3 F1 versus 72.8 for a matched prompted judge; this is a verifier comparison, not evidence of lower inference cost.
Key insights
- High-quality generative process reward models can be trained with orders of magnitude less step-level supervision than discriminative PRMs, achieving comparable or superior performance.
- Rigorous, process-based filtering of synthetically generated verification Chain-of-Thoughts is crucial for mitigating issues like invalid outputs and overthinking in generative verifiers.
- Fine-tuning large reasoning models on even a small, high-quality synthetic dataset dramatically enhances their reliability, reduces undesirable inference behaviors, and improves F1-score for process-level verification.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.