Reproduce “Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiwei Chen, Yuguang Yao, Yihua Zhang, Bingquan Shen, Gaowen Liu, Sijia Liu
The paper shows that supervised safety fine-tuning can make vision-language models associate words with refusal labels, creating a “safety mirage.” It tests changing “What” and “Share,” and proposes machine unlearning to remove harmful-example influence while retaining data. On LLaVA-1.5-7B, RMU-Unlearning lowers ASR from 54.98% to 10.18% and RR from 91.76% to 7.56% versus Mixed-SFT on VLGuard; results are benchmark-specific.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.