Reproduce “Fast-MIA: Efficient and Scalable Membership Inference for LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hiromu Takahashi, Shotaro Ishihara
Fast-MIA is a Python library that accelerates Membership Inference Attack (MIA) evaluations for Large Language Models (LLMs). It achieves up to a 19.5x speedup for complex black-box attacks and an overall 5x speedup by leveraging high-throughput batch inference and a cross-method caching architecture, while accurately replicating the performance of existing MIA methods.
Key insights
- Significant speedups in MIA evaluation can be achieved through optimized inference and intelligent caching of intermediate results without compromising the accuracy of attack performance metrics.
- Leveraging high-throughput inference engines and cross-method caching is critical for scaling MIA evaluations to modern LLMs and complex, computationally intensive attack strategies.
- A standardized framework and configuration system greatly enhance the reproducibility, extensibility, and fair comparative analysis of diverse MIA methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.