Reproduce “N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alex Foote, Neel Nanda, Esben Kran, Ionnis Konstas, Fazl Barez
N2G automates the interpretation of individual neuron representations in Large Language Models, converting their contextual activation patterns into measurable graph structures. The method quantifies interpretation quality and effectively visualizes polysemantic neuron behaviors, establishing a scalable and verifiable approach to mechanistic interpretability, particularly for earlier model layers.
Key insights
- Quantifiable metrics (precision, recall, F1-score) can be applied to assess the accuracy of neuron interpretations generated by automated methods.
- Neuron behavior complexity tends to increase with model depth; earlier layers exhibit simpler, more easily graph-interpretable functions compared to deeper layers.
- The graph-based approach can inherently represent and visually distinguish polysemantic behaviors of a single neuron through disconnected subgraphs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.