Reproduce “HelpSteer2-Preference: Complementing Ratings with Preferences” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, Yi Dong
NVIDIA researchers introduced HelpSteer2-Preference, a high-quality dataset enabling the first rigorous comparison of Bradley-Terry and Regression reward model training paradigms. Their work identified a synergistic approach that combines these methods with a novel Scaled Bradley-Terry loss and ExPO, achieving a 94.1 score on RewardBench and enabling an aligned Llama-3.1-70B-Instruct model to rank #1 on Arena Hard with an 85.0 score.
Key insights
- When provided with adequately matched, high-quality data, optimal configurations of Bradley-Terry and Regression-style RMs perform comparably on overall benchmarks, challenging prior assumptions of one paradigm's inherent superiority.
- Combining Regression-style initialization with Scaled Bradley-Terry training and ExPO extrapolation yields a synergistic effect, indicating that both rating and preference data contain complementary information for more robust RMs.
- Online RLHF algorithms like REINFORCE and PPO consistently outperform offline DPO variants, with REINFORCE achieving superior alignment by leveraging a stable, unbiased Monte-Carlo baseline for reward maximization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.