Reproduce “Innocence in the Crossfire: Roles of Skip Connections in Jailbreaking Visual Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Palash Nandi, Maithili Joshi, Tanmoy Chakraborty
Researchers at IIT Delhi systematically investigated how Visual Language Models (VLMs) can be jailbroken, identifying specific prompt design factors and introducing `SKIP-CON`, a novel internal architectural modification. Their work reveals that VLMs exhibit increased susceptibility to generating inappropriate content, particularly when exposed to implicitly toxic memes, with `SKIP-CON` substantially enhancing attack success rates.
Key insights
- VLM's ability to distinguish between benign and harmful content significantly degrades in multimodal contexts compared to unimodal processing.
- Internal architectural manipulations, specifically injecting early-stage features via `SKIP-CON`, substantially increase VLM jailbreak success rates.
- Implicitly toxic content, such as harmful memes, can be as effective as explicitly toxic visuals in eliciting inappropriate VLM outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.