Reproduce “Deep Reinforcement Learning with Robust and Smooth Policy” on OpenResearch

About this paper

Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, Tuo Zhao

This research introduces the Smoothness Regularized Reinforcement Learning (SR²L) framework, which embeds a smoothness constraint into deep RL algorithms to address issues of sample inefficiency and training instability. The proposed method improves learning efficiency and reliability while inherently enhancing the robustness of policies against state measurement errors, as demonstrated by higher cumulative rewards and reduced performance variance across various continuous control tasks.

Key insights

• Optimal policies in continuous state spaces should ideally be smooth, meaning similar states should lead to similar actions, aligning with inherent environmental properties.
• The inherent non-smoothness of deep neural networks can be a root cause for sample inefficiency, training instability, and vulnerability to measurement errors in deep reinforcement learning.
• Imposing a smoothness regularization on the policy or value function, formulated as an adversarial optimization problem, can simultaneously improve sample efficiency, training stability, and robustness to state perturbations.

Summary by alphaXiv.