Reproduce “An Information Theoretic Approach to Machine Unlearning” on OpenResearch

About this paper

Jack Foster, Kyle Fogarty, Stefan Schoepf, Zack Dugue, Cengiz Öztireli, Alexandra Brintrup

A zero-shot unlearning method, Just in Time (JiT) unlearning, was developed based on an information-theoretic approach to efficiently remove data influence from trained models. It achieves competitive performance with other state-of-the-art methods while being over 5 times faster than previous zero-shot approaches, particularly in scenarios lacking access to the original training data.

Key insights

• A data sample's influence on a model can be approximated by the information gain it provided during training, which can be inferred from the local geometry (curvature) of the model's decision boundary.
• Inducing local smoothing around forgotten data points is an effective strategy for unlearning in a zero-shot setting, as it removes the specific influence without requiring retain data.
• Unlearning can be achieved by increasing the entropy of the model's output distributions over the forget set, making the model less confident about forgotten data, similar to a retrained model.

Summary by alphaXiv.