Reproduce “SafeTy Reasoning Elicitation Alignment for Multi-Turn Dialogues” on OpenResearch

About this paper

Martin Kuo, Jianyi Zhang, Aolin Ding, Louis DiValentin, Amin Hass, Benjamin F Morris, Isaac Jacobson, Randolph Linderman and 7 more

A collaborative effort between Duke University and Accenture introduces STREAM, a plug-and-play defense mechanism that leverages a safety reasoning moderator trained on a human-annotated, metacognitive Chain-of-Thought dataset. This system significantly reduces Attack Success Rate (ASR) against multi-turn jailbreaking attacks by up to 51.2% for models like GPT-4.1, while preserving the target LLM's functional capabilities.

Key insights

• Infusing a safety reasoning moderator with "metacognitive Chain-of-Thought" capabilities enables it to detect subtle and evolving malicious intent within multi-turn dialogues.
• A flexible, plug-and-play external moderator can provide robust defense against jailbreaking across diverse LLMs (both open and closed-source) without compromising their core functional capabilities.
• High-quality, human-annotated datasets containing detailed safety reasoning and Chain-of-Thought explanations are essential for training effective multi-turn safety mechanisms.

Summary by alphaXiv.