Reproduce “Defining and Characterizing Reward Hacking” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David Krueger

Researchers at the University of Oxford, Mila, and the University of Cambridge provide the first formal definition of reward hacking, termed 'hackability,' within reinforcement learning. Their theoretical analysis demonstrates that imperfect proxy reward functions are almost always hackable in broad, continuous policy spaces, yet can be unhackable for finite policy sets under specific conditions.

Key insights

  • If an AI agent can explore a broad, continuous range of behaviors (i.e., the policy set contains an 'open subset'), then a proxy reward function can only be 'unhackable' if it perfectly reproduces the true reward function's preference ordering over policies.
  • For finite sets of policies (e.g., all deterministic policies in a finite MDP), it is always possible to find non-trivial proxy reward functions that are 'unhackable' with respect to a true reward function.
  • The concept of 'simplification' (an asymmetric form of unhackability where the proxy collapses distinctions of the true reward) is possible under specific dimensionality conditions related to the policy visit counts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.