Reproduce “Blackbox Model Provenance via Palimpsestic Membership Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rohith Kuditipudi, Jing Huang, Sally Zhu, Diyi Yang, Christopher Potts, Percy Liang
Researchers at Stanford University developed a statistical framework to prove the provenance of blackbox large language models and their generated text by leveraging a phenomenon called "palimpsestic memorization." The methods provide statistically rigorous detection with provable false positive control across both direct query and observational settings, demonstrated effectively on various Pythia, OLMo, and TinyStories models.
Key insights
- LLMs exhibit "palimpsestic memorization," meaning their internal representations and output probabilities are measurably influenced by the *order* in which training examples were processed, particularly for later-seen data.
- By assuming a randomized training shuffle, provenance can be statistically proven through an independence test, offering provable control over false positive errors (Type-I errors).
- Employing an independent reference model in query-based tests significantly reduces the required amount of interaction with a blackbox model while maintaining high statistical power.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.