Reproduce “Exploring Membership Inference Vulnerabilities in Clinical Large Language Models” on OpenResearch

About this paper

Alexander Nemecek, Zebin Yun, Zahra Rahmani, Yaniv Harel, Vipin Chaudhary, Mahmood Sharif, Erman Ayday

An empirical study by researchers at Case Western Reserve University and Tel Aviv University reveals that clinical Large Language Models (LLMs) exhibit measurable privacy leakage through membership inference attacks, even when queries are semantically similar but linguistically varied. The study found that such models' confidence differences, indicative of membership, are robust to reasonable linguistic variation, highlighting a vulnerability relevant to real-world clinical use.

Key insights

• Clinical LLMs possess a non-zero privacy signal, confirming measurable leakage through membership inference attacks, even if limited.
• Membership inference signals are robust to semantic paraphrasing; attacks with lexically varied but semantically equivalent queries achieve comparable performance to baseline attacks.
• Llemr's privacy exposure primarily arises from global confidence patterns across entire sequences rather than highly localized memorization of specific rare tokens.

Summary by alphaXiv.