Reproduce “Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling” on OpenResearch

About this paper

Alex Nikulkov

Temporally Coherent Reward Modeling (TCRM) enhances large language model reward models by training them to make intermediate token outputs predictive of the final reward. This approach improves interpretability and enables process-level evaluation without additional human labeling, while also allowing the model to function as both a reward and value function in PPO, leading to a 27% reduction in peak GPU memory and 19% faster training.

Key insights

• Reward model intermediate outputs can be made predictive of the final reward's conditional expectation by adding specific regularization terms during training, even using only outcome-level preference data.
• The proposed regularization terms (Lookahead Consistency and Smoothness) directly correspond to classical reinforcement learning value functions (Monte Carlo and Temporal Difference learning).
• This structural correspondence allows a single TCRM to efficiently serve as both the reward model and the value model within the PPO framework.

Summary by alphaXiv.