Reproduce “Spectral Unlearning: Transformer Structure-Preserving Updates for Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sung Il Choi, Junhao Cai, Dohun Kim, Changhee Joo
The authors introduce spectral unlearning, which updates transformer weight matrices jointly rather than changing scalar entries independently, while leaving the unlearning objective unchanged. On TOFU’s Llama-3.2-1B-Instruct Forget 5% benchmark, SU-NPO-SOAP scores 0.67 aggregate, versus 0.54 for NPO-AdamW and 0.72 for the retain-trained target. This balanced benchmark score does not establish complete deletion or retraining equivalence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.