Reproduce “LoRA is All You Need for Safety Alignment of Reasoning LLMs” on OpenResearch

About this paper

Yihao Xue, Baharan Mirzasoleiman

Researchers at the University of California, Los Angeles demonstrate that applying Low-Rank Adaptation (LoRA) during supervised finetuning on refusal datasets effectively performs safety alignment for reasoning-capable LLMs. This method preserves reasoning abilities while achieving strong safety, mitigating the typical performance degradation known as the "Safety Tax."

Key insights

• Safety-related behavior in reasoning LLMs can be effectively induced by modifying weights within a low-dimensional subspace, suggesting that high-rank updates are unnecessary and detrimental to established reasoning skills.
• LoRA's effectiveness is maximized when the finetuning objective (safety) is intrinsically low-dimensional and the base capability (reasoning) is intrinsically high-dimensional, making it robust to practical rank choices.
• Specific LoRA configurations, such as using rank-1 updates, targeting MLP up-projection layers, and applying changes to middle layers of the model, are particularly efficient for balancing safety and reasoning.

Summary by alphaXiv.