Reproduce “Towards Understanding the Influence of Reward Margin on Preference Model Performance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bowen Qin, Duanyu Feng, Xi Yang
Researchers from BAAI and Sichuan University introduce a method to incorporate 'reward margin' into preference models for reinforcement learning from human feedback (RLHF), improving reward model accuracy by up to 6.31% on benchmarks. The approach estimates preference differences without requiring costly explicit human margin labels, enhancing the practical utility of reward models in selecting high-quality responses.
Key insights
- Higher-accuracy reward models inherently exhibit a broader distribution of reward margins (higher mean, lower kurtosis, positive skewness), indicating they naturally differentiate preferences more distinctly.
- Human preferences are not merely binary but possess diverse magnitudes, as evidenced by GPT-4's ability to assign varying margin labels, a nuance often missed by traditional reward models.
- Even without explicit human-annotated margins, an automatically estimated 'reward confidence' (local approximation of reward differences within a batch) can effectively guide reward model training to capture preference magnitudes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.