Reproduce “Shadow Unlearning: A Neuro-Semantic Approach to Fidelity-Preserving Faceless Forgetting in LLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dinesh Srivasthav P, Ashok Urlana, Rahul Mishra, Bala Mallikarjunarao Garlapati, Ponnurangam Kumaraguru

This research introduces Shadow Unlearning, a framework for removing specific information from Large Language Models using only anonymized data. Its Neuro-Semantic Projector Unlearning (NSPU) method achieves effective forgetting while preserving model utility, enhancing privacy, and demonstrating high computational efficiency.

Key insights

  • Semantic relationships between original and anonymized data can be established at the neural activation level, enabling unlearning without re-identifying individuals.
  • Targeted forgetting can be achieved by projecting activations onto a low-dimensional 'forget subspace,' effectively attenuating specific knowledge with minimal impact on overall model utility.
  • Unlearning performance can be significantly improved by leveraging data heterogeneity, as diverse domains act as a regularizer, leading to more modular representations that are easier to selectively unlearn.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.