Reproduce “Wasserstein Robust Reinforcement Learning” on OpenResearch

About this paper

Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, Jun Wang

Researchers primarily from Huawei R&D UK developed Wasserstein Robust Reinforcement Learning (WR²L), an algorithm that addresses the sim-to-real gap by formulating robust RL as a min-max game with Wasserstein constraints on environmental dynamics. The system consistently achieved superior generalization performance across varying continuous control tasks in MuJoCo environments, leveraging a scalable zero-order optimization method for black-box simulators.

Key insights

• Using the Wasserstein distance provides a theoretically sound and geometrically sensitive measure for defining and constraining uncertainty in continuous dynamics, leading to more meaningful robustness.
• Directly perturbing parameters of differential equation solvers or simulators (rather than using deep networks for dynamics) ensures that learned adversarial dynamics remain physically valid.
• The zero-order optimization approach effectively handles black-box simulators and high-dimensional dynamics parameter spaces, which is crucial for practical application where explicit model gradients are unavailable.

Summary by alphaXiv.