Reproduce “Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lin Zhang, Lijie Hu, Di Wang
A new framework, Self-Influence Circuit Analysis Framework (SICAF), combines circuit analysis with self-influence to reveal the layer-wise reasoning pathways within Transformer models. Applying SICAF to GPT-2 on the Indirect Object Identification task uncovered a human-interpretable, hierarchical information processing flow.
Key insights
- The Self-Influence Circuit Analysis Framework (SICAF) can effectively combine circuit identification with self-influence to map nuanced, layer-wise reasoning paths in Transformers.
- Critical circuits responsible for specific tasks are remarkably small, often comprising only 1-2% of a model's total edges, while retaining high performance fidelity.
- Language models exhibit a consistent, human-interpretable hierarchical reasoning process, with early layers identifying entities, middle layers interpreting actions, and final layers inferring conclusions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.