Reproduce “Robust Reinforcement Learning using Adversarial Populations” on OpenResearch

About this paper

Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, Alexandre Bayen

Researchers at the University of California, Berkeley, including Pieter Abbeel and Alexandre Bayen, developed Robustness via Adversary Populations (RAP), a method that trains reinforcement learning agents against multiple adversaries simultaneously to create more generally robust policies. The approach demonstrated that agents trained against a single adversary are susceptible to exploitation by other adversaries, while RAP achieved superior performance on out-of-distribution tasks, such as a 238% improvement over vanilla PPO on the Hopper test set.

Key insights

• Agents trained against a single adversary are prone to overfitting, making them highly exploitable by other adversaries (even those within the same class), failing to achieve true generalized robustness.
• Training an agent with a population of diverse adversaries forces the agent to learn a more general, unexploitable policy that is robust to a wider range of perturbations.
• The effectiveness of Domain Randomization is highly dependent on expert-defined parameter ranges, and 'bad' parameterizations can lead to policies that are brittle and perform poorly in parts of the expected operating range.

Summary by alphaXiv.