Reproduce “Focal Reward: Balanced Reinforcement Learning under Rubric-Based Rewards” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu Huang, Zihua Zhao, Zhaoxin Huan, Wanli Gu, Feng Hong, Xinmu Ge, Lin Yuan, Weichang Wu and 4 more
Focal Reward introduces a dynamic reweighting mechanism for rubric-based rewards in large language model (LLM) alignment, addressing imbalanced optimization by estimating criterion-level saturation. The method consistently improved average scores on various benchmarks by up to 2.50 points and enhanced performance on initially harder criteria, without significantly increasing computational cost.
Key insights
- Dynamically adjusting reward weights based on criterion-level saturation effectively balances optimization across diverse rubric criteria.
- Estimating saturation from the 'frontier' of high-performing rollouts provides a robust signal for identifying criteria with the most remaining improvement potential.
- Reallocating training emphasis towards under-satisfied criteria leads to more targeted and efficient policy updates, improving overall model adherence to complex instructions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.