Reproduce “AdvCLIP: Downstream-agnostic Adversarial Examples in Multimodal Contrastive Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziqi Zhou, Shengshan Hu, Minghui Li, Hangtao Zhang, Yechao Zhang, Hai Jin
Researchers at Huazhong University of Science and Technology developed AdvCLIP, a framework to create downstream-agnostic adversarial examples for multimodal contrastive learning models like CLIP. This method achieved up to an 80.05% attack success rate on cross-modal retrieval and an average accuracy drop of over 55% in image classification tasks, demonstrating a general vulnerability of these models even against common defenses.
Key insights
- Achieving robust, downstream-agnostic adversarial attacks against cross-modal models requires disrupting not just individual feature embeddings but also the topological relationships between samples in the feature space.
- Common defense mechanisms such as corruption, pruning, and adversarial training are largely ineffective against the universal adversarial examples generated by AdvCLIP.
- Transformer-based cross-modal architectures (e.g., ViT) show greater vulnerability to these attacks compared to CNN-based architectures like ResNet.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.