Reproduce “Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learning” on OpenResearch

About this paper

Lama Alssum, Hani Itani, Hasan Abed Al Kader Hammoud, Philip Torr, Adel Bibi, Bernard Ghanem

Researchers from KAUST and the University of Oxford systematically apply continual learning (CL) methods to preserve the safety alignment of Large Language Models (LLMs) during fine-tuning, demonstrating that techniques like Dark Experience Replay (DER) can significantly reduce attack success rates (ASR) to under 6% even with poisoned data, while maintaining task utility.

Key insights

• Safety degradation in fine-tuned LLMs closely mirrors catastrophic forgetting, indicating that established continual learning principles can be directly applied to address this issue.
• Memory-based continual learning methods, particularly Dark Experience Replay (DER), offer a superior balance of safety preservation and downstream task utility compared to other CL paradigms and existing safety baselines.
• While most CL methods reduce safety degradation, their robustness varies significantly with the severity of harmful data; some techniques (e.g., LwF) become brittle under high poison ratios, highlighting the need for context-aware defense strategies.

Summary by alphaXiv.