Reproduce “LLMs Can Unlearn Refusal with Only 1,000 Benign Samples” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yangyang Guo, Ziwei Xu, Si Liu, Zhiming Zheng, Mohan Kankanhalli

Researchers at NUS and Beihang University developed a "refusal unlearning" technique that compromises LLM safety alignment by fine-tuning models with as few as 1,000 benign samples, causing them to generate harmful content after initiating a refusal prefix. This method reduced safety scores by an average of 50-60% across 16 diverse LLMs, including commercial models, by exploiting the superficiality of current refusal mechanisms.

Key insights

  • LLM safety alignment based on prefix-driven refusal mechanisms is highly vulnerable to adversarial unlearning, indicating a "shallow alignment" rooted in token memorization rather than robust reasoning.
  • Compromising LLM safety does not require harmful data or complex prompt engineering; it can be achieved stealthily with a minimal set of benign samples (as few as 1,000).
  • The effectiveness of refusal unlearning is positively correlated with the "refusal strength" of the chosen prefix, suggesting that stronger refusal cues in training data lead to more effective unlearning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.