Reproduce “Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned” on OpenResearch

About this paper

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez and 28 more

Anthropic researchers conducted an extensive human-in-the-loop red teaming effort with 38,961 adversarial attacks on large language models (LLMs) to systematically identify and mitigate harmful outputs. Their findings indicate that Reinforcement Learning from Human Feedback (RLHF) models become more resistant to red teaming as they scale, while simpler prompting methods demonstrate limited efficacy in adversarial settings.

Key insights

• Adversarial red teaming provides a more stringent evaluation of language model safety than static evaluations, revealing that simple prompting methods are insufficient against determined human adversaries.
• Even with advanced safety interventions like RLHF and Rejection Sampling, models can still produce highly harmful outputs, indicating a "lower tail" of persistent vulnerabilities.
• The process of assessing red team attack success is subjective, with low to fair inter-annotator agreement on what constitutes a successful elicitation of harm.

Summary by alphaXiv.