Reproduce “Poisoning Language Models During Instruction Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alexander Wan, Eric Wallace, Sheng Shen, Dan Klein
Researchers at UC Berkeley demonstrated that instruction-tuned large language models are highly susceptible to data poisoning during fine-tuning, where a small number of malicious examples can induce widespread, stealthy misclassification or degenerate outputs across numerous unrelated tasks. This work reveals that larger models can be more vulnerable to certain poisoning types, challenging conventional scaling assumptions.
Key insights
- Poisoning effects exhibit strong cross-task generalization, causing systemic errors on numerous held-out tasks not directly targeted or poisoned during training.
- Larger models demonstrate an "inverse scaling" trend for polarity poisoning, becoming substantially more susceptible to attacks than smaller models, swiftly overwriting vast pre-training knowledge.
- Data poisoning can be remarkably stealthy, often not significantly impacting the model's accuracy on regular inputs without the trigger phrase, making detection challenging.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.