Reproduce “Policy Smoothing for Provably Robust Reinforcement Learning” on OpenResearch

About this paper

Aounon Kumar, Alexander Levine, Soheil Feizi

This research introduces Policy Smoothing, a method extending randomized smoothing to Reinforcement Learning (RL) to provide provable robustness against adaptive adversarial perturbations on observations. It establishes theoretical guarantees for the agent's total expected reward over an entire episode, demonstrating non-vacuous certificates that can surpass the performance of undefended agents under attack in several benchmark environments.

Key insights

• For adaptive adversaries, the worst-case attack can be simplified to a deterministic adversary that uses its entire budget to perturb only the first dimension of the first observation vector.
• Randomized smoothing, typically applied to static classification, can be rigorously extended to provide provable robustness certificates for the cumulative performance (total reward) of RL agents in dynamic environments.
• The L2 robustness guarantee for an RL agent under adaptive attack can be expressed using the standard normal CDF, providing a tractable and tight lower bound on expected total reward.

Summary by alphaXiv.