Reproduce “Reasoning as an Adaptive Defense for Safety” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Taeyoun Kim, Fahim Tajwar, Aditi Raghunathan, Aviral Kumar
Carnegie Mellon University researchers developed TARS (Training Adaptive Reasoners for Safety), a reinforcement learning approach that enables large language models to adaptively reason about safety using chain-of-thought traces, leading to superior safety-refusal trade-offs and enhanced robustness against adversarial attacks.
Key insights
- Lightweight SFT is crucial for inducing diverse and exploratory reasoning traces, which are more conducive to effective refinement through subsequent reinforcement learning.
- Mixing harmful, harmless, and ambiguous prompts with separate reward functions is essential to prevent models from learning degenerate refusal strategies and to maintain their reasoning abilities.
- TARS-trained models develop distinct internal representations, enabling better separation of harmful and harmless prompts, and adaptively allocate more computational effort (longer reasoning) to ambiguous safety queries.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.