Reproduce “Forgetting-MarI: LLM Unlearning via Marginal Information Regularization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shizhou Xu, Yuan Ni, Stefan Broecker, Thomas Strohmer

Forgetting-MarI presents an information-theoretic framework for LLM unlearning, focusing on removing only the marginal influence of specific data while preserving overall model utility. It demonstrates robust performance in continual unlearning scenarios and renders unlearned content empirically undetectable by state-of-the-art white-box detectors.

Key insights

  • The core insight is that effective unlearning should precisely remove the *marginal effect* of specific data, not its 'full information,' to avoid degrading general model utility and preserving shared knowledge.
  • Mutual information, particularly JSD, provides a precise and theoretically grounded metric for quantifying this marginal information, directly linking its minimization to the undetectability of unlearned data.
  • Integrating marginal information regularization into a standard fine-tuning objective leads to stable training dynamics and a robust utility-unlearning trade-off, outperforming methods that rely on conflicting objectives.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.