Reproduce “Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization” on OpenResearch

About this paper

Xiyue Peng, Hengquan Guo, Jiawei Zhang, Dongqing Zou, Ziyu Shao, Honghao Wei, Xin Liu

Researchers from ShanghaiTech University, SenseTime Research, and Washington State University introduced Rectified Policy Optimization (RePO), an algorithm that enforces critical safety constraints on large language models (LLMs) to prevent individual unsafe responses. RePO achieved a 96.08% model-based safety rate for Alpaca-7B while maintaining helpfulness, significantly improving upon baselines that rely on average safety.

Key insights

• Relying solely on expected safety constraints in LLM alignment is insufficient, as it permits 'safety compensation' where models can be statistically safe but still produce critically unsafe individual outputs.
• Enforcing per-instance 'critical safety constraints' is essential for robust and trustworthy LLM behavior, preventing the generation of harmful content on a case-by-case basis.
• Optimizing under strict per-instance safety requirements can be effectively achieved through a rectified re-formulation, which selectively penalizes only actual safety violations, coupled with a dynamic penalty mechanism.

Summary by alphaXiv.