Reproduce “Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models” on OpenResearch

About this paper

Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, Chun-Ying Huang

Researchers from National Yang Ming Chiao Tung University and IBM Research developed Safe LoRA, a data-free and training-free method to preserve Large Language Model safety alignment during parameter-efficient fine-tuning. This approach significantly reduces harmfulness scores and attack success rates across various fine-tuning scenarios, often outperforming existing defenses while maintaining task utility.

Key insights

• The difference in weights between aligned and unaligned base models (the "alignment matrix") can capture a safety-critical subspace, which can then be used to guide future fine-tuning updates.
• Conditionally projecting LoRA weight updates post-fine-tuning allows for a balance between preserving safety alignment and maintaining the model's utility for its intended task.
• A computationally faster, approximate projection method (C_i) derived from the alignment matrix proves as effective, and sometimes more so, than a mathematically precise but slower method, enhancing practical applicability.

Summary by alphaXiv.