Reproduce “Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions” on OpenResearch

About this paper

Ziqing Lu, Babak Hassibi, Lifeng Lai, Weiyu Xu

Researchers propose a novel reinforcement learning framework that allows agents to actively modify their environment's underlying dynamics through "model-changing actions." This framework, instantiated as Multi-Level Configurable Time-Varying MDPs, enables agents to learn optimal policies not just within a fixed world, but also to reconfigure it for improved long-term returns.

Key insights

• Agents can achieve substantially higher cumulative rewards by actively shaping their environment through model-changing actions, rather than being confined to passive adaptation.
• The problem of learning optimal model-changing actions can be formalized as a higher-order MDP, where the lower-level environment's transition kernels become states for an upper-level configuring agent.
• Theoretical analysis provides rigorous error bounds, quantifying the propagation of errors from imperfect model estimation and configuration inaccuracies throughout the multi-level learning process.

Summary by alphaXiv.