Reproduce “Categorizing Variants of Goodhart's Law” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
David Manheim, Scott Garrabrant
Manheim and Garrabrant categorize distinct failure modes arising from optimizing systems based on proxy metrics, clarifying the mechanisms of metric-induced goal misalignment often referred to as Goodhart's Law. This work establishes a structured taxonomy of Regressional, Extremal, Causal, and Adversarial Goodhart effects, providing a refined framework for understanding and addressing unintended consequences in fields like AI alignment.
Key insights
- Goodhart effects arise from distinct mechanisms, ranging from inherent statistical noise and out-of-distribution behavior to direct causal intervention and strategic responses from other agents.
- Even under ideal conditions, optimizing for a proxy leads to 'Regressional Goodhart' where the true goal is predictably lower than the proxy when the proxy is high (regression to the mean).
- Distinguishing true Causal Goodhart, which involves altering causal paths, from situations where mistaken causal assumptions lead to worsened Regressional or Extremal effects, is critical for analysis.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.