Reproduce “The Pen is Mightier than the Pixel: Natural Textual Unlearnable Example for Large Vision Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenyu Yi, Jiawen Zong, Sami Hormi, Bingquan Shen, Xudong Jiang, Alex Chichung Kot
The paper proposes NTUE, which changes image captions to disrupt identity recognition during model fine-tuning. Its decoupling attack replaces the pictured person’s name with another identity, leaving pixels unchanged. On LLaVA-1.5 7B fine-tuned with Trump images, NTUE(DA) produced 0% identification accuracy with 20 poisoned samples, versus 96.5% for clean data; adding PGD yielded 54.5%. This result concerns that identity task, not general privacy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.