Reproduce “Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg-Kirkpatrick, Reza Shokri

The authors introduce a likelihood-ratio attack comparing a target model’s score for a clinical note with a reference model’s, accounting for text difficulty. In the ClinicalBERT-Base test, using PubMed-BERT as reference and held-out MIMIC-III notes as nonmembers, it achieves AUC 0.900 versus 0.662 for target-model loss; at 10% false-positive rate, recall is 79.2% versus 15.6%. This shows stronger membership inference here, not universally.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.