Reproduce “Measuring Forgetting of Memorized Training Examples” on OpenResearch

About this paper

Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song and 3 more

Researchers from Google and several universities quantified how machine learning models forget training examples over time, demonstrating that this 'passive forgetting' naturally reduces their susceptibility to privacy attacks like membership inference and data extraction across image, speech, and language models. The study found that forgetting is primarily driven by the inherent stochasticity of training, with example repetition being the most critical factor influencing its speed.

Key insights

• Random sampling (stochasticity) within training processes like SGD is a primary mechanism that causes models to forget, as demonstrated theoretically for mean estimation and empirically for neural networks.
• Non-convexity, while often associated with deep learning, can paradoxically prevent forgetting in certain scenarios, as shown with a constructed K-means example where an outlier is persistently memorized.
• The most critical factor influencing forgetting speed is the number of times an example is repeated in the training data, with highly duplicated or 'hard' examples forgotten much more slowly.

Summary by alphaXiv.