Reproduce “Probing Language Models for Pre-training Data Detection” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenhua Liu, Tong Zhu, Chuanyuan Tan, Haonan Lu, Bing Liu, Wenliang Chen
Researchers from Soochow University and OPPO AI Center developed a white-box probing technique that analyzes internal model activations to determine if specific texts were part of a Large Language Model's pre-training data. This method consistently outperformed existing baselines on both established and a new, more challenging benchmark, ArxivMIA, offering a more robust approach to data detection.
Key insights
- Internal activations within LLMs encode distinct and reliable signals indicating whether a particular text was encountered during pre-training, offering a deeper detection mechanism than external metrics.
- Fine-tuning an LLM on a specific dataset effectively alters its internal representations in a manner consistent with pre-training, enabling the generation of labeled activation data for probe training.
- The complexity and rarity of text content, as shown by the ArxivMIA benchmark, directly impact the difficulty of detecting pre-training data membership, with academic abstracts proving more challenging than general text.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.