Reproduce “Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models” on OpenResearch

About this paper

Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu and 1 more

Researchers from Alibaba Group and The Chinese University of Hong Kong developed Equilibrate RLHF, a framework that balances helpfulness and safety in large language models by optimizing safety data distribution and selectively highlighting safety-critical tokens. This approach achieved superior safety alignment with up to 13% higher Bal-Safe scores on LLAMA3-8B-instruct compared to DPO with larger datasets, while maintaining or improving general helpfulness.

Key insights

• Simply increasing the quantity of safety data does not uniformly improve safety and often leads to an "over-safe" state rather than a "truly safe" one.
• Different types of safety data (EHD, IHD, MHD) have distinct impacts on model alignment, with IHD being more effective for malicious intent and EHD for knowledge-based risks.
• By highlighting safety-critical tokens, the AMA approach enables LLMs to better understand the underlying causes of risks, leading to more nuanced and constructive safe responses beyond simple refusals.

Summary by alphaXiv.