Reproduce “On Evaluating the Durability of Safeguards for Open-Weight LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr and 2 more
The authors examine how reliably two weight-based safeguards, RepNoise and Tamper-Resistant Safeguards, withstand fine-tuning attacks. They rerun attacks across random seeds, vary training implementations and evaluation prompts, and assess utility and safety side effects. On TAR-Bio-v2, the highest WMDP-Bio accuracy exceeds 60% in all three configurations. These findings concern only two fine-tuning threat models, not all weight modifications.
Key insights
- Evaluating durable LLM safeguards against adversarial modifications is inherently difficult and prone to misinterpretation, with subtle aspects of the evaluation setup drastically altering perceived robustness.
- Approaches claiming to "unlearn" or "remove" harmful information often merely suppress it, making the knowledge recoverable even with benign or unrelated fine-tuning data.
- Current safeguards, while attempting to address specific safety objectives, may inadvertently lead to significant regressions in overall model utility or even exacerbate other safety risks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.