Reproduce “Applying sparse autoencoders to unlearn knowledge in language models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eoin Farrell, Yeu-Tong Lau, Arthur Conmy
Researchers at the ML Alignment & Theory Scholars (MATS) Program and Arthur Conmy investigated using sparse autoencoders (SAEs) to remove undesirable knowledge, such as bioweapons information, from language models. The method demonstrated the feasibility of targeted knowledge suppression but currently incurs more side effects on general capabilities compared to fine-tuning approaches, while revealing that negative scaling of feature activations is crucial for unlearning.
Key insights
- Negative scaling or clamping of feature activations is necessary for effective unlearning; simple zero-ablation is ineffective, suggesting an active 'push' away from the concept is required.
- Targeted unlearning using individual bio-related Sparse Autoencoder (SAE) features is feasible with minimal side effects on general knowledge.
- The importance of SAE decoder vectors in unlearning varied across different model architectures and intervention layers, suggesting the unlearning mechanism might be context-dependent.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.