Reproduce “Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecification” on OpenResearch

About this paper

Thomas Kwa, Drake Thomas, Adrià Garriga-Alonso

This paper investigates the effectiveness of KL divergence regularization in RLHF under reward misspecification, demonstrating theoretically that if reward modeling errors are heavy-tailed, policies can achieve arbitrarily high proxy reward without increasing true utility, a phenomenon termed "catastrophic Goodhart." Empirical analysis of current open-source language model reward functions suggests they are light-tailed, while synthetic experiments confirm these theoretical predictions.

Key insights

• When reward modeling errors are heavy-tailed, KL divergence regularization fails to prevent policies from achieving arbitrarily high proxy rewards while incurring minimal KL divergence from the base model, yet yielding no true utility gain.
• Conversely, if reward errors are light-tailed and independent of true utility, KL regularization can effectively drive optimization towards higher true utility.
• The tail behavior of the error distribution is a critical factor, and this dependency is not exclusive to KL divergence, also appearing in conditioning-based optimization models.

Summary by alphaXiv.