Reproduce “TAME: Test-Time Adversarial Prompt Tuning via Mixture-of-Experts for Vision-Language Models” on OpenResearch

About this paper

Xin Wang, Yixu Wang, Jiaming Zhang, Ruofan Wang, Jiaqi Yu, Kai Chen, Jingjing Chen, Xingjun Ma and 1 more

TAME introduces a test-time adversarial prompt tuning method for Vision-Language Models (VLMs) that uses a Multimodal Mixture-of-Prompts (MMoP) architecture to enhance zero-shot adversarial robustness. It dynamically composes expert prompts based on input, achieving superior defense against attacks while preserving clean accuracy without task-specific retraining.

Key insights

• Integrating a Mixture-of-Experts architecture into prompt tuning significantly enhances the expressive capacity and adaptability of test-time defenses against diverse adversarial perturbations.
• Combining multi-view entropy minimization with adversarial-clean feature alignment is crucial for achieving high adversarial robustness while concurrently preserving zero-shot clean accuracy.
• Independently modeling visual and textual prompt experts (VLI design) provides superior flexibility for correcting adversarial distortions compared to joint or visual-only prompt designs.

Summary by alphaXiv.