Reproduce “Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wenxuan Wang, Kuiyi Gao, Youliang Yuan, Jen-tse Huang, Qiuzhi Liu, Shuai Wang, Wenxiang Jiao, Zhaopeng Tu
Researchers from Renmin University of China, Chinese University of Hong Kong, Johns Hopkins University, and Hong Kong University of Science and Technology, in collaboration with Tencent, developed the "Chain-of-Jailbreak" (CoJ) attack, demonstrating how leading commercial image generation models can be coerced into producing harmful content through multi-turn, step-by-step editing. They also introduced "Think-Twice Prompting," an effective defense strategy, significantly increasing safety against such iterative attacks.
Key insights
- Current safety mechanisms in state-of-the-art image generation models are largely ineffective against multi-turn, iterative editing attacks, as they fail to track cumulative malicious intent.
- Malicious content can be generated even when direct, single-prompt queries are initially refused, exposing a critical vulnerability in how safety filters operate across conversational contexts.
- Encouraging internal reflection and explicit safety assessment (e.g., "Think-Twice Prompting") within the model can dramatically improve its robustness against sophisticated jailbreaking attempts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.