Reproduce “Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hua Farn, Hsuan Su, Shachi H Kumar, Saurav Sahay, Shang-Tse Chen, Hung-yi Lee

A model merging strategy effectively safeguards large language models from safety degradation after fine-tuning on downstream tasks, maintaining or improving task performance without requiring additional safety data or training. This approach reduces the Attack Success Rate (ASR) on harmful instructions while simultaneously enhancing task-specific metrics across various LLMs and domains.

Key insights

  • Model merging provides a data-agnostic and computationally efficient solution to effectively address safety degradation in fine-tuned LLMs.
  • Linear interpolation allows for a controllable trade-off, enabling the simultaneous improvement of both safety (reduced ASR) and task-specific utility.
  • The merging technique not only preserves safety alignment but also helps restore other lost capabilities of the original aligned model, such as instruction-following.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.