Reproduce “Do Multimodal RAG Systems Leak Data? A Comprehensive Evaluation of Membership Inference and Image Caption Retrieval Attacks” on OpenResearch

About this paper

Ali Al-Lawati, Suhang Wang

Researchers at The Pennsylvania State University empirically evaluated privacy vulnerabilities in multimodal Retrieval-Augmented Generation (mRAG) systems by conducting Membership Inference (MIA) and Image Caption Retrieval (ICR) attacks. They found that mRAG systems are susceptible to both attacks, achieving up to 0.993 F1 for exact image MIAs and over 68% exact-match for exact image ICRs, even under common image transformations.

Key insights

• mRAG pipelines are fundamentally vulnerable to both MIA and ICR attacks, even when input images are subjected to common transformations, demonstrating a persistent leakage risk.
• Architectural and configuration choices, such as the strategic ordering of query images within the prompt context and the presence of reranking, can influence the extent of privacy leakage.
• Preliminary mitigation attempts using external LLMs to identify malicious prompts showed limited effectiveness, particularly for membership inference attempts, suggesting a need for more sophisticated defense mechanisms.

Summary by alphaXiv.