Reproduce “Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark Study” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
DongGeon Lee, Joonwon Jang, Jihae Jeong, Hwanjo Yu
Researchers at Pohang University of Science and Technology introduced MEMESAFETYBENCH, a benchmark using 50,430 real meme images to assess Vision-Language Model safety. Their study found that VLMs are significantly more vulnerable to meme-based harmful prompts than to artificial inputs, with larger models generally showing improved safety, though multi-turn interactions offer only partial mitigation.
Key insights
- VLMs demonstrate a consistently higher vulnerability to harmful instructions when real meme images are present, indicating that memes are more effective at eliciting unsafe responses than artificial inputs.
- While multi-turn conversational contexts offer some mitigation against harmful meme-based prompts, they do not fully eliminate vulnerabilities, leaving models more susceptible than in text-only scenarios.
- Increasing VLM model scale generally correlates with improved safety, characterized by higher refusal rates and fewer harmful responses, although task completion rates show a more complex relationship with size.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.