Reproduce “Process Reinforcement through Implicit Rewards” on OpenResearch

About this paper

Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He and 17 more

PRIME trains an implicit process reward model online from outcome labels on policy rollouts, without separately labeled reasoning steps. It derives token-level rewards from the model’s probability differences from a reference model and combines them with outcome rewards in reinforcement learning. After 240 steps, PRIME scores 50.6% on AMC, versus 47.0% for outcome-only RLOO, a test-accuracy comparison.

Key insights

• Online updating of the Implicit PRM is crucial for sustained performance gains and to prevent reward overoptimization, as static reward models lead to diminishing returns.
• The Implicit PRM can be effectively initialized directly from a Supervised Fine-Tuned (SFT) or even a base model, circumventing the need for a dedicated, costly reward model training phase.
• Implicit PRMs function more effectively as reward models for advantage estimation rather than as value models, providing direct, fine-grained feedback to the policy.

Summary by alphaXiv.