Reproduce “Behemoth: Benchmarking Unlearning in LLMs Using Fully Synthetic Data” on OpenResearch

About this paper

Eugenia Iofinova, Dan Alistarh

Researchers at IST Austria developed "Behemoth," a fully synthetic data generation framework that allows precise control over data distribution and tokenization to benchmark unlearning in Large Language Models. Their work demonstrates that low-rank updates can be effective for specific edits, but unlearning entire relationships or editing implied knowledge proves challenging even in controlled settings.

Key insights

• LLMs struggle to robustly infer and modify implied relationships in nested data structures, suggesting that factual knowledge is stored largely independently rather than through generalized inference.
• Completely 'forgetting' an entire relationship by remapping all its objects is considerably more difficult than changing specific facts, requiring substantial model alterations and impacting other knowledge.
• The optimal choice of layers for model editing is highly dependent on the task, and interpretability techniques like activation patching may not always align with the most effective fine-tuning strategies for preserving overall model quality during edits.

Summary by alphaXiv.