Reproduce “Why Some Models Resist Unlearning: A Linear Stability Perspective” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wei-Kai Chang, Rajiv Khanna
Researchers at Purdue University developed a theoretical framework for machine unlearning, demonstrating how the linear stability of optimization dynamics and a novel "mix-coherence measure" predict a model's ability to forget data. The work surprisingly shows that models exhibiting higher memorization, often due to low signal-to-noise data, can be easier to unlearn.
Key insights
- The success or failure of gradient-based unlearning can be predicted by the linear stability of the model's parameters around the original minimum, governed by a 'mix-coherence measure'.
- A counter-intuitive relationship exists where models that exhibit stronger memorization (e.g., of low signal-to-noise data) can be paradoxically easier to unlearn.
- High coherence between retain and forget sets leads to resistance to unlearning, as updates for the forget set interfere significantly with the retain set's influence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.