Reproduce “Robust Reinforcement Learning on State Observations with Learned Optimal Adversary” on OpenResearch

About this paper

Huan Zhang, Hongge Chen, Duane Boning, Cho-Jui Hsieh

This paper presents a method for training robust Deep Reinforcement Learning agents that can withstand adversarial perturbations to their state observations. It achieves this by learning an "optimal" empirical adversary through standard reinforcement learning and then using an alternating training framework to iteratively improve both the agent's policy and the adversary's attack strategy. Experiments on continuous control tasks demonstrate that agents trained with this approach achieve significantly higher robustness against strong adversarial attacks, particularly when using history-dependent policies.

Key insights

• For a fixed agent policy, finding an optimal adversary can be reformulated as solving an equivalent standard Markov Decision Process (MDP) for the adversary.
• When the adversary is fixed, the State-Adversarial MDP (SA-MDP) is equivalent to a Partially Observable Markov Decision Process (POMDP), suggesting that history-dependent policies are beneficial for robustness.
• A comprehensive defense strategy against adversarial observations benefits from addressing both intrinsic policy weakness and function approximator vulnerability.

Summary by alphaXiv.