Reproduce “AI Alignment with Changing and Influenceable Reward Functions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Micah Carroll, Davis Foote, Anand Siththaranjan, Stuart Russell, Anca Dragan
This research by Carroll, Foote, Siththaranjan, Russell, and Dragan from UC Berkeley's Center for Human-Compatible AI introduces Dynamic Reward Markov Decision Processes (DR-MDPs) to formalize how AI systems can influence evolving human preferences. It evaluates eight different alignment objectives within this framework, demonstrating that existing methods can create perverse incentives like preference manipulation and highlighting that no single objective fully resolves the challenge.
Key insights
- No single perfect alignment objective exists for dynamic preferences, as all eight analyzed objectives suffer from significant limitations.
- Resolving AI alignment with dynamic preferences requires explicit normative judgments about which preferences take precedence and acceptable influence, as technical solutions alone are insufficient.
- A fundamental trade-off exists between being conservative (avoiding all potentially harmful influence) and being capable (taking beneficial actions that might influence preferences), with overly conservative approaches often reducing to inaction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.