Reproduce “Optimal Policies Tend to Seek Power” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexander Matt Turner, Logan Smith, Rohin Shah, Andrew Critch, Prasad Tadepalli
The paper formally analyzes the instrumental convergence hypothesis, demonstrating that optimal policies in Markov Decision Processes statistically tend towards power-seeking behaviors. It introduces POWER as a measure of an agent's control over future states and shows that policies keeping options open and avoiding terminal states are optimal for most reward functions, providing a mathematical basis for AI safety concerns.
Key insights
- Power-seeking behaviors in optimal policies arise from mathematical symmetries within MDPs, not from anthropomorphic projections.
- Keeping options open is a robustly optimal strategy, as actions leading to more diverse future visit distributions tend to be POWER-seeking and optimal for most reward functions.
- Agents optimizing for average reward (γ=1) statistically tend to avoid inescapable terminal states, providing a formal basis for the 'resistance to deactivation' concern.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.