Reproduce “MCA-Bench: A Multimodal Benchmark for Evaluating CAPTCHA Robustness Against VLM-based Attacks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zonglin Wu, Yule Xue, Yaoyao Feng, Xiaolong Wang, Yiren Song
Researchers from Southwest University and the National University of Singapore introduced MCA-Bench, a comprehensive multimodal benchmark evaluating CAPTCHA robustness against VLM-based attacks. The benchmark, which includes over 180,000 samples across 20 tasks and four modalities, revealed that while fine-tuned VLMs achieve over 96% accuracy on simpler CAPTCHAs, their performance drops to as low as 2.5% on tasks demanding physical interaction or multi-step reasoning, consistently remaining below human performance levels.
Key insights
- Visual confusion, interaction depth, and semantic complexity are critical factors that jointly increase CAPTCHA resistance against VLM-based attacks.
- Fine-tuning VLMs on diverse CAPTCHA datasets yields substantial performance gains, enabling models to outperform zero-shot baselines and some closed-source peers.
- A persistent gap remains between VLM and human performance on tasks requiring multi-step reasoning, nuanced contextual understanding, or precise interactive behaviors, despite VLM advancements.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.