Reproduce “Safety of Multimodal Large Language Models on Images and Texts” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xin Liu, Yichen Zhu, Yunshi Lan, Chao Yang, Yu Qiao
A systematic survey maps the current landscape of safety for Multimodal Large Language Models (MLLMs) on images and text, consolidating research on evaluation, attack, and defense strategies. It identifies unique vulnerabilities introduced by the visual modality, including susceptibility to adversarial image attacks, OCR-driven jailbreaks, and weakened textual safety alignment.
Key insights
- The visual modality introduces new, cost-effective attack vectors such as adversarial image perturbations and OCR-driven jailbreaks, which can bypass textual safety filters.
- Cross-modal training in MLLMs can paradoxically weaken the pre-existing safety alignment of their underlying text-only LLM components, increasing susceptibility to textual adversarial attacks.
- Current MLLM safety evaluation datasets and metrics are diverse but often limited in scope, lacking comprehensive and finer-grained safety capability partitioning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.