Reproduce “GRAM-R$^2$: Self-Training Generative Foundation Reward Models for Reward Reasoning” on OpenResearch

About this paper

Chenglong Wang, Yongyu Mu, Hang Zhou, Yifu Huo, Ziming Zhu, Jiali Zeng, Murun Yang, Bei Li and 5 more

GRAM-R2 presents a self-training methodology for generative foundation reward models, enabling them to incorporate explicit reward reasoning without requiring expensive human-annotated rationales. This approach leverages large amounts of unlabeled data, leading to superior performance in LLM alignment tasks and efficient adaptation across domains.

Key insights

• Explicit reward reasoning can be effectively instilled in foundation models through a self-training process that synthesizes rationales from a dedicated Preference-Proving Model.
• Leveraging large-scale unlabeled data alongside rationale-free labeled data significantly reduces dependence on expensive human annotations, enabling scalable development of generalist reward models.
• Incorporating reward reasoning enhances the model's generalization capabilities and robustness against reward overoptimization, leading to more stable and reliable LLM alignment.

Summary by alphaXiv.