Reproduce “Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching” on OpenResearch

About this paper

Aleksandar Makelov, Georg Lange, Neel Nanda

The authors show that subspace activation patching can change model behavior without identifying a feature faithfully. They explain how a direction can mix input-correlated variation with a normally inactive, output-affecting route. In GPT-2 Small indirect-object-identification tests, the layer-8 MLP patch yields 46.7% FLDD, versus 13.5% after removing its nullspace component. GPT-2 XL factual-recall patches show a similar pattern in the tested settings.

Key insights

• Observing a causal effect from subspace activation patching does not always imply a faithful understanding of the model's normal computation; it can activate dormant parallel pathways that are not causally active under normal conditions.
• The "MLP-in-the-middle illusion" is likely prevalent in LLMs due to the linear-algebraic structure of transformer blocks, suggesting that many apparent "circuit components" found in MLPs could be artifacts of this illusion.
• Rank-1 model editing methods (like ROME) can succeed by leveraging these illusory, dormant pathways, rather than truly modifying existing knowledge-storing mechanisms in specific layers.

Summary by alphaXiv.