Reproduce “Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Liang Chen, Xueting Han, Li Shen, Jing Bai, Kam-Fai Wong
Researchers from The Chinese University of Hong Kong, Microsoft Research Asia, and Sun Yat-sen University developed Vulnerability-Aware Alignment (VAA) to mitigate safety degradation in Large Language Models caused by harmful fine-tuning. VAA identifies data vulnerabilities and uses Group Distributionally Robust Optimization to proactively reduce harmful scores by an average of 12.4% compared to standard fine-tuning, while preserving model utility.
Key insights
- Uneven forgetting in LLMs during harmful fine-tuning is data-dependent, with specific alignment examples being consistently more susceptible to compromise.
- These data vulnerability patterns are transferable across different fine-tuning tasks and LLM architectures, allowing for a practical proxy-based estimation.
- Vulnerable examples exhibit lower robustness to weight perturbations, indicated by steeper loss gradients, explaining their higher susceptibility to forgetting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.