Reproduce “DCMI: A Differential Calibration Membership Inference Attack Against Retrieval-Augmented Generation” on OpenResearch

About this paper

Xinyu Gao, Xiangtao Meng, Yingkai Dong, Zheng Li, Shanqing Guo

Researchers at Shandong University developed DCMI, a membership inference attack exposing privacy vulnerabilities in Retrieval-Augmented Generation (RAG) systems by accurately determining if specific data samples are part of their retrieval databases. The attack employs a differential calibration technique to overcome noise from non-member documents, achieving up to 94.35% accuracy against RAG systems and demonstrating practical success on platforms like Dify and MaxKB.

Key insights

• The "interference of non-member-retrieved documents" is a critical, previously unaddressed factor that significantly hampers RAG MIA performance, which DCMI effectively mitigates through differential calibration.
• Member-retrieved documents exhibit high sensitivity to subtle query perturbations, causing a distinct drop in RAG confidence that can be exploited for accurate membership inference.
• RAG systems, including commercial-grade platforms, possess inherent privacy vulnerabilities that allow for practical membership inference, highlighting an urgent need for privacy-aware design.

Summary by alphaXiv.