Reproduce “Localizing Model Behavior with Path Patching” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, Aryaman Arora
Redwood Research introduces Path Patching, a method to precisely locate deep neural network behaviors within specific computational paths by formalizing localization claims and evaluating them quantitatively with an Average Unexplained Effect (AUE) metric. The technique enabled refining explanations for induction heads in transformers and showed more specific localization than causal tracing on GPT-2 for number prediction tasks.
Key insights
- Path patching formalizes localization claims, moving interpretability from qualitative descriptions to quantitative, falsifiable hypotheses.
- Operating on paths, rather than just nodes, allows for a more expressive and fine-grained representation of information flow and specific interventions.
- The Average Unexplained Effect (AUE) provides a clear, quantitative metric to evaluate how well a hypothesized set of paths explains a behavior, enabling rigorous comparison.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.