Reproduce “Representation-Guided Parameter-Efficient LLM Unlearning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zeguan Xiao, Lang Mo, Yun Chen, Lei Yang, Jiehui Zhao, Lili Yang, Guanhua Chen
ReGLU, a parameter-efficient unlearning framework, utilizes representation-guided initialization and orthogonal regularization to more effectively remove specific knowledge from Large Language Models. This approach improves unlearning quality on benchmarks like TOFU and WMDP while preserving model utility, addressing the forget-retain trade-off by leveraging geometric properties of representation spaces.
Key insights
- Instead of targeting individual polysemantic parameters, knowledge can be more effectively disentangled and targeted for unlearning by leveraging the geometric properties of representation subspaces.
- By aligning LoRA update matrices with subspaces that maximize differential variance between forget and retain sets, a more precise and effective initial unlearning direction can be established.
- Explicitly constraining LoRA updates to be orthogonal to critical directions of retained knowledge representations minimizes interference with the model's general capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.