Reproduce “Your Policy Regularizer is Secretly an Adversary” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Grégoire Delétang, Markus Kunesch, Shane Legg, Pedro Ortega

The authors interpret policy regularization in reinforcement learning as a game against an adversary that changes rewards within a set defined by the regularizer. For KL regularization, the reference policy shapes the allowed shifts. They prove that, for a fixed policy, every reward in its robust set yields expected reward at least the regularized objective, a guarantee limited to that set.

Key insights

  • Policy regularization intrinsically functions as an agent implicitly hedging against worst-case reward perturbations chosen by an adversary, framing regularized RL as a minimax game.
  • For optimal policies, the worst-case reward perturbation aligns precisely with the advantage function for any "-divergence, generalizing existing connections to optimality conditions.
  • Properly interpreting entropy regularization as a special case of divergence regularization resolves prior ambiguities, confirming that stronger regularization increases robustness to reward perturbations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.