Reproduce “Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training” on OpenResearch

About this paper

Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He and 2 more

This work theoretically demonstrates that reward model inaccuracies in the high-reward tail are the primary cause of reward over-optimization in Large Language Model (LLM) post-training. It introduces an iterative Refinement-through-Differentiation (RTD) workflow for constructing rubric-based reward models, leveraging diverse high-quality off-policy data to improve accuracy in distinguishing excellent from merely great responses, thereby enhancing LLM alignment and mitigating over-optimization.

Key insights

• Reward model misspecification within the high-reward tail, specifically the inability to reliably differentiate truly excellent from very good responses, is the primary theoretical driver of reward over-optimization.
• Rubric-based reward models, when carefully constructed, can effectively leverage off-policy data to improve high-reward tail accuracy, proving more robust than traditional preference models in this context.
• An iterative refinement process centered on differentiating diverse, high-quality responses (RTD) is crucial for developing sophisticated rubrics that prevent over-optimization and sustain LLM performance over extended training.

Summary by alphaXiv.