Reproduce “CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning” on OpenResearch

About this paper

Biao Yi, Tiansheng Huang, Baolei Zhang, Tong Li, Lihai Nie, Zheli Liu, Li Shen

Researchers at Nankai University and Sun Yat-sen University developed CTRAP, a defense mechanism that protects LLMs from harmful fine-tuning attacks by inducing controlled model collapse when malicious adaptation is detected. The system successfully maintains lower harmful scores during fine-tuning while preserving the model's benign functionality, demonstrating a new approach to LLM safety.

Key insights

• CTRAP introduces a paradigm shift from selective knowledge removal to conditional model collapse, accepting controlled degradation as a defense mechanism against harmful fine-tuning.
• Embedding a 'collapse trap' during initial model alignment allows for conditional activation, where the model's fundamental language processing abilities degrade only when malicious adaptation is detected.
• The approach effectively maintains model utility for legitimate applications while significantly hindering the acquisition of harmful capabilities through fine-tuning.

Summary by alphaXiv.