Reproduce “Reward Model Ensembles Help Mitigate Overoptimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Thomas Coste, Usman Anwar, Robert Kirk, David Krueger

This study investigates ensemble reward models with conservative optimization strategies to mitigate overoptimization in Reinforcement Learning from Human Feedback (RLHF). The research demonstrates that these methods improve policy alignment and performance, effectively countering overoptimization even with noisy human labels and various model scales.

Key insights

  • Ensemble-based conservative optimization effectively mitigates overoptimization and significantly improves policy performance across different policy optimization methods and model scales.
  • The benefits of using reward model ensembles are orthogonal to the gains from scaling up reward model size or training dataset size, allowing for synergistic combination.
  • The approach demonstrates robustness to label noise, various ensemble cardinalities, and the uncertainty penalty hyperparameter in UWO, making it practical for real-world RLHF deployments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.