Reproduce “Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning” on OpenResearch

About this paper

Tan Jing, Xiaorui Li, Chao Yao, Xiaojuan Ban, Yuetong Fang, Renjing Xu, Zhaolin Yuan

Researchers at the University of Science and Technology Beijing and the Hong Kong University of Science and Technology (Guangzhou) developed Adaptive Scaling of Policy Constraints (ASPC), a framework that automatically adjusts the policy constraint scale in offline reinforcement learning. This method results in policies with a 35% average improvement in normalized performance over a TD3+BC baseline across D4RL benchmarks, while eliminating the need for manual hyperparameter tuning.

Key insights

• The optimal scale for policy constraints is dynamic and highly dependent on the quality of the dataset and the specific task, requiring continuous adjustment during training.
• A bilevel optimization approach with second-order gradients can effectively and automatically adapt this critical constraint scale, leading to more stable learning and higher policy performance across diverse settings.
• The inclusion of specific regularization terms ($L_2$ for Q-value stability and $L_3$ for BC loss variation) is crucial, as they mutually constrain changes in policy and Q-values, enabling stable convergence across various domains.

Summary by alphaXiv.