Reproduce “Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuyou Zhang, Miao Li, William Han, Yihang Yao, Zhepeng Cen, Ding Zhao
Carnegie Mellon University researchers developed RATIONAL, a reasoning-enhanced fine-tuning framework that integrates explicit safety reasoning into LLMs. This approach significantly improves robustness against sophisticated jailbreak attacks, achieving 0/135 ASR on specific SorryBench categories, while also enhancing helpfulness and interpretability by enabling context-aware safety decisions.
Key insights
- Explicit, interpretable reasoning can serve as a fundamental, intrinsic mechanism for LLM safety, moving beyond superficial refusal patterns to dynamic, context-aware decision-making.
- Training models with both adversarial (rejection) and benign (compliance) reasoning rationales is crucial for effectively balancing safety robustness with helpfulness and mitigating over-refusal.
- The safety improvements gained from reasoning-enhanced fine-tuning generalize well to unseen attack types and diverse evaluation benchmarks, indicating a deeper internalization of safety principles.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.