Reproduce “E-MIA: Exam-Style Black-Box Membership Inference Attacks against RAG Systems” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zelin Guan, Shengda Zhuo, Zeyan Li, Jinchun He, Wangjie Qiu, Zhiming Zheng, Shuqiang Huang

E-MIA tests whether a candidate document is included in a retrieval-augmented generation (RAG) knowledge base through ordinary queries and answers. It turns document-specific facts into four types of graded questions, then combines scores. On a TREC-COVID evaluation with Llama 3.1 8B, it achieved a true-positive rate of 0.982 at a 0.005 false-positive rate, but answers may have other sources.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.