Reproduce “Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minseok Choi, Kyunghyun Min, Jaegul Choo
The LINGTEA framework enables cross-lingual unlearning in multilingual language models, selectively erasing specific information across various languages while robustly preserving the model's general utility and performance. It consistently lowers memorization accuracy on forget sets while maintaining competitive perplexity and general knowledge on retained data.
Key insights
- Unlearning information in a high-resource language does not automatically transfer to other languages within the same multilingual model, revealing a significant cross-lingual vulnerability.
- An adaptive weighting scheme, which adjusts the influence of a teacher model based on its confidence in specific languages, enhances both unlearning effectiveness and the preservation of model utility in multilingual contexts.
- Simultaneously applying unlearning techniques across a randomly sampled subset of languages provides a practical and efficient solution for multilingual models, bypassing the prohibitive cost of sequential unlearning or full model retraining.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.