Reproduce “RL with KL penalties is better viewed as Bayesian inference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tomasz Korbak, Ethan Perez, Christopher L Buckley

This paper re-interprets Reinforcement Learning (RL) with Kullback-Leibler (KL) penalties, as used in RLHF for large language models, demonstrating its mathematical equivalence to variational Bayesian inference. The work shows that the KL-regularized RL objective is proportional to the negative KL divergence between the fine-tuned LM and a specific Bayesian posterior, thereby explaining why KL regularization prevents distribution collapse.

Key insights

  • The empirical success of KL-regularized RL in LM alignment stems from its implicit solution to a Bayesian inference problem, rather than its explicit formulation as an RL problem focused on reward maximization.
  • KL penalties are not merely heuristics but serve as a principled mechanism to prevent distribution collapse by ensuring the fine-tuned LM approximates a well-defined Bayesian posterior.
  • A Bayesian perspective allows for a clear separation between the "modelling problem" (defining the desired target distribution) and the "inference problem" (approximating or sampling from it), offering a more systematic approach to LM alignment.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.