Reproduce “Concept Unlearning in Large Language Models via Self-Constructed Knowledge Triplets” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tomoya Yamashita, Yuuki Yamanaka, Masanori Yamada, Takayuki Miura, Toshiki Shibahara, Tomoharu Iwata

Researchers at NTT developed Concept Unlearning (CU), a self-supervised method for Large Language Models to forget abstract concepts rather than individual data points. The approach uses the LLM itself to generate knowledge triplets and explanatory sentences about the target concept, achieving complete unlearning while preserving general utility and unrelated knowledge.

Key insights

  • Concept Unlearning is achievable by having the LLM dynamically identify and structure its own knowledge for forgetting, removing reliance on external data or predefined targets.
  • The combination of triplet-based and sentence-based loss functions is crucial for achieving both comprehensive conceptual forgetting and high selectivity, preserving non-target knowledge.
  • Improving the coverage and precision of the self-generated knowledge graph leads to more effective and selective unlearning, enhancing the preservation of the model's general utility.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.