Reproduce “Robust Deep Reinforcement Learning through Adversarial Loss” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tuomas Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, Tsui-Wei Weng
The RADIAL-RL framework enables efficient training of Deep Reinforcement Learning (DRL) agents to be robust against lp-norm adversarial attacks on observations. It consistently achieves superior robustness and computational efficiency compared to prior methods across diverse DRL algorithms and environments, alongside introducing an improved evaluation metric.
Key insights
- Integrating inexpensive robustness verification bounds, like those from IBP, directly into the DRL loss function provides a principled and computationally efficient pathway to robust DRL training.
- Tailoring the adversarial loss formulation, specifically by using a weighted margin based on Q-value differences, can prevent undesirable Q-value overestimation bias seen in other robust DRL methods.
- The proposed Greedy Worst-Case Reward (GWC) metric offers an efficient and empirically validated approximation of true worst-case reward, addressing the computational challenges of robust DRL evaluation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.