Reproduce “Targeted Recovery of Weight-Space Mechanisms From Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Antoine Vigouroux, Lee Sharkey
Targeted Parameter Decomposition (tPD) introduces an efficient method to identify and isolate specific computational mechanisms within neural network weight matrices by training on both target and non-target data streams. This approach enables precise unlearning and knowledge editing with minimal collateral effects, making mechanistic interpretability more scalable for larger models.
Key insights
- Weight-space mechanisms can be efficiently localized to specific data subsets by leveraging a two-stream training process and adversarial ablation, significantly reducing computational overhead.
- Mechanistic faithfulness, crucial for interpretable interventions, can be maintained by ensuring that isolated components do not spuriously activate on non-target data or cause unintended side effects when modified.
- A sparse set of weight-space subcomponents can be identified that accurately represent specific model behaviors or stored knowledge, enabling granular control over model function.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.