Reproduce “A Study of the Attention Abnormality in Trojaned BERTs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weimin Lyu, Songzhu Zheng, Tengfei Ma, Chao Chen
Researchers from Stony Brook University and IBM Research identified "attention focus drifting" as a key mechanism through which Trojan attacks manipulate BERT models, leading to the development of AttenTD, an attention-based detector that outperforms existing methods in identifying compromised NLP models.
Key insights
- "Attention focus drifting," characterized by attention weights concentrating on trigger tokens, is a prevalent and specific abnormality in Trojaned BERT models.
- This drifting behavior, particularly in deeper layers and surprisingly in separator heads, is causally linked to the Trojan's misclassification effect, as demonstrated by head pruning experiments.
- The trigger token gains dominant attribution in drifting attention heads, indicating its disproportionate influence on the model's final prediction in compromised BERTs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.