Reproduce “Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Roberto-Rafael Maura-Rivero, Chirag Nagpal, Roma Patel, Francesco Visin
Researchers at Google DeepMind and Google Research developed the Inada Reward Transformation (IRT) to improve large language model (LLM) alignment during Reinforcement Learning from Human Feedback (RLHF). This method, inspired by economic utility functions, significantly increased harmlessness by 75% in a strict Win Rate against a baseline while preserving helpfulness and reducing reward hacking behaviors.
Key insights
- Principles from economic utility theory can be effectively applied to design more principled and robust reward aggregation schemes in AI, particularly for enhancing LLM alignment.
- Disproportionately penalizing critically low reward values (e.g., for harmlessness) is crucial for improving overall model alignment and preventing superficial or 'punting' behaviors.
- Diminishing returns for already high reward values help prevent over-optimization of satisfactory attributes, allowing the model to focus resources on improving underperforming areas.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.