Reproduce “Adaptive Preference Scaling for Reinforcement Learning with Human Feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ilgee Hong, Zichong Li, Alexander Bukharin, Yixiao Li, Haoming Jiang, Tianbao Yang, Tuo Zhao
Adaptive Preference Scaling (APS) enhances Reinforcement Learning from Human Feedback by introducing instance-specific weights for human preferences, moving beyond fixed-scale reward modeling. This approach significantly improves policy performance in robotic control and natural language generation tasks and addresses the misalignment between reward model accuracy and downstream policy effectiveness.
Key insights
- The theoretically derived instance-specific scaling factor (τᵢ) adaptively assigns smaller reward differences for ambiguous preferences and larger differences for clear preferences, as validated by theoretical propositions.
- The adaptive scaling method fundamentally mitigates the misalignment between reward model selection criteria (preference accuracy) and actual downstream policy performance.
- The approach is grounded in Distributionally Robust Optimization, providing a principled and robust theoretical framework for handling uncertainty in human feedback.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.