Reproduce “Attention Slipping: A Mechanistic Understanding of Jailbreak Attacks and Defenses in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaomeng Hu, Pin-Yu Chen, Tsung-Yi Ho
A study from The Chinese University of Hong Kong and IBM Research revealed "Attention Slipping," a phenomenon where large language models systematically reduce attention to unsafe content during jailbreak attacks, and introduced "Attention Sharpening," an efficient defense that enhances safety without incurring additional computational overhead.
Key insights
- "Attention Slipping" is a universal phenomenon observed across diverse jailbreak attacks, where LLMs systematically reduce their attention to harmful content as an attack progresses and succeeds.
- Existing query perturbation-based defenses (e.g., Token Highlighter, SmoothLLM) implicitly mitigate Attention Slipping by partially restoring the model's focus on unsafe prototypes.
- Directly manipulating the attention mechanism through Attention Sharpening can effectively counteract jailbreak dynamics, forcing the model to re-engage with or be overwhelmed by the unsafe prompt.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.