Reproduce “Invariant Policy Optimization: Towards Stronger Generalization in Reinforcement Learning” on OpenResearch

About this paper

Anoopkumar Sonar, Vincent Pacelli, Anirudha Majumdar

Anoopkumar Sonar and colleagues at Princeton University developed Invariant Policy Optimization (IPO), an algorithm that applies causal invariance principles to reinforcement learning to create policies that generalize across diverse environments. The method consistently outperformed standard policy gradient techniques like PPO, showing improved performance on unseen test domains in LQR, a colored-key grid-world, and a simulated robotic door opening task, notably learning more robust manipulation strategies.

Key insights

• Generalization in RL can be improved by learning policies that exploit the "causes of successful actions" and ignore task-irrelevant environmental changes.
• Causal invariance principles, previously applied in supervised learning, can be effectively adapted to guide policy learning in reinforcement learning.
• Reframing the invariant policy learning as a game among domain-specific policies encourages convergence to a robust, shared policy representation.

Summary by alphaXiv.