Reproduce “BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIP” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, Wei Liu
Researchers from Tsinghua University and Tencent Data Platform developed BadCLIP, a backdoor attack targeting CLIP models during their prompt learning adaptation phase with limited data. It employs a trigger-aware mechanism where an image-based trigger dynamically influences text features, achieving attack success rates often exceeding 99% while preserving clean accuracy and generalizing across diverse datasets and domains.
Key insights
- A critical security vulnerability exists in CLIP's prompt learning phase, allowing effective backdoor injection with limited downstream data.
- Generalizable backdoor attacks on multi-modal models like CLIP are achievable by designing a trigger that dynamically influences both image and text feature spaces simultaneously.
- The collaboration between trigger-influenced image features and dynamically generated, trigger-aware text prompts is essential for high attack success rates and cross-modal generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.