Reproduce “Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Johannes Ackermann, Takashi Ishida, Masashi Sugiyama
This research from The University of Tokyo and RIKEN AIP introduces Off-Policy Corrected Reward Modeling (OCRM), a method that addresses the overoptimization problem in Reinforcement Learning from Human Feedback (RLHF) for large language models by using importance weighting to consistently adapt the reward model to the evolving policy's distribution, achieving notable improvements in alignment performance without requiring new human feedback.
Key insights
- Standard RLHF leads to an inconsistent reward model (RM) estimation because the policy's evolving data distribution diverges from the initial data used to train the RM, causing overoptimization.
- Off-Policy Corrected Reward Modeling (OCRM) uses importance weighting to consistently estimate RM parameters under the current policy's distribution, even with fixed training data.
- Combining an adaptively updated RM with changes to KL regularization and optimizer resets is essential for enabling the policy to effectively learn from the corrected reward signal.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.