Reproduce “Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhibin Duan, Guowei Rong, Zhuo Li, Bo Chen, Mingyuan Zhou, Dandan Guo

The Bayesian Non-Negative Reward Model (BNRM) introduces a principled framework for reward modeling in RLHF, integrating Bayesian principles with non-negative factor analysis to mitigate reward over-optimization. This approach enhances robustness to noise and misgeneralization, yielding up to 13.3% improvement in HHH Alignment accuracy and significantly reducing reward bias towards superficial features like response length.

Key insights

  • Sparsity and non-negativity constraints on latent factors promote disentangled and interpretable reward representations, allowing for a mechanistic understanding of reward signals.
  • Local sparsity in the θ factors encourages instance-level disentanglement, activating only semantic features relevant to a specific response and suppressing spurious ones.
  • Global sparsity in the Φ dictionary acts as a population-level regularizer, identifying and down-weighting systematic, non-causal biases across the dataset (e.g., response length bias).

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.