Reproduce “Knowledge Swapping via Learning and Unlearning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingyu Xing, Lechao Cheng, Shengeng Tang, Yaxiong Wang, Zhun Zhong, Meng Wang
This research introduces "Knowledge Swapping," a new task enabling deep learning models to simultaneously acquire new knowledge, retain existing knowledge, and selectively forget specified information. The approach utilizes Low-Rank Adaptation (LoRA) and sparse regularization, demonstrating effective forgetting (near 0% accuracy/mIoU on forgotten data) while maintaining high performance on learned and retained data across various computer vision tasks.
Key insights
- An empirical analysis revealed a "knock-on feature hierarchy" where learning predominantly updates low-level to high-level features, while forgetting acts in the reverse direction, from high-level to low-level features.
- The "Learning Before Forgetting" (L F) strategy is crucial for robust knowledge swapping, as learning first establishes low-level representations that are less disturbed by subsequent high-level forgetting.
- LoRA combined with Lasso regularization efficiently achieves selective knowledge management, allowing targeted parameter updates for learning and forgetting while preserving the integrity of other model knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.