Reproduce “SmoothGuard: Defending Multimodal Large Language Models with Noise Perturbation and Clustering Aggregation” on OpenResearch

About this paper

Guangzhi Su, Shuchang Huang, Yutong Ke, Zhuohang Liu, Long Qian, Kaizhu Huang

SmoothGuard, a lightweight and model-agnostic defense framework, enhances the robustness of Multimodal Large Language Models (MLLMs) against adversarial attacks by combining randomized noise perturbation with clustering aggregation. The approach significantly reduces adversarial attack success rates (e.g., average ASR decreased from 0.0345 to 0.0183 for Qwen2.5-VL-7B) while largely preserving model utility on benign tasks.

Key insights

• Randomized noise perturbation, when combined with clustering-based aggregation, can effectively filter out adversarial influences on MLLM outputs without requiring extensive model retraining.
• There is an optimal range for noise perturbation levels (e.g., a Gaussian noise standard deviation of 0.1-0.2) that achieves a balance between substantial robustness gains and minimal degradation in model utility.
• A model-agnostic defense can be effectively generalized to different MLLM architectures and evaluated systematically using a modified adversarial image generation strategy on HuggingFace models.

Summary by alphaXiv.