Reproduce “Uncovering and Aligning Anomalous Attention Heads to Defend Against NLP Backdoor Attacks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haotian Jin, Yang Li, Haihui Fan, Lin Shen, Xiangfang Li, Bo Li
Researchers from the Chinese Academy of Sciences and UCAS introduce a defense framework that leverages anomalous attention head similarity to detect and mitigate backdoor attacks in Large Language Models. The method significantly reduces attack success rates across various attacks and models while maintaining high performance on clean tasks, operating without requiring trigger knowledge or a clean reference model.
Key insights
- Backdoored models exhibit unusually high cosine similarity in attention patterns among certain heads when activated by a trigger, indicating a uniform, malicious response.
- Combining a head's influence (gradient sensitivity) with its attention pattern similarity allows for accurate identification of malicious attention components.
- Targeted alignment of only suspicious heads to the 'clean' attention distributions of safe heads effectively suppresses backdoor behavior without corrupting benign parts of the model.
- Differentiated learning rates during fine-tuning enable efficient recovery of clean task performance while preventing backdoor reactivation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.