Reproduce “The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, Dawn Song
The paper introduces exposure, a measure of whether a generative sequence model favors a randomly inserted canary over alternatives from the same format. Researchers vary canary repetitions, compare log-perplexities, and use probability-guided search to test extraction. Models with equal final accuracy showed no extraction below exposure 31 and extraction in every trial above 33 in one controlled experiment, not universally.
Key insights
- Unintended memorization is a persistent issue in generative neural networks, occurring early in training and resisting common regularization methods, suggesting it's distinct from traditional overfitting.
- The introduced 'exposure' metric effectively quantifies memorization, directly correlating with the practical extractability of secret sequences from a trained model.
- Differentially Private Stochastic Gradient Descent (DP-SGD) is uniquely effective in eliminating unintended memorization, significantly outperforming other evaluated defense mechanisms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.