Reproduce “Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers” on OpenResearch

About this paper

Liang Lin, Zhihao Xu, Xuehai Tang, Shi Liu, Biyu Zhou, Fuqing Zhu, Jizhong Han, Songlin Hu

The Chinese Academy of Sciences and Renmin University of China developed the Paper Summary Attack (PSA), a method that embeds harmful queries within summaries of LLM safety research papers to bypass model safeguards. This approach achieved high attack success rates against state-of-the-art LLMs and revealed alignment biases where models exhibit differing vulnerabilities based on whether the input context is related to attack or defense strategies.

Key insights

• LLMs exhibit "diametrically opposed vulnerability biases," meaning their susceptibility to jailbreaking varies significantly based on whether the contextual academic paper discusses attack or defense strategies.
• Current LLM safety mechanisms misinterpret the intent of harmful queries when they are embedded within authoritative academic contexts, leading to the generation of positive or neutral sentiment tokens internally instead of activating safety guardrails.
• The very knowledge designed to improve LLM safety (academic research on attacks and defenses) can be weaponized as a vector for vulnerability.

Summary by alphaXiv.