Reproduce “Analyzing Transformers in Embedding Space” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guy Dar, Mor Geva, Ankit Gupta, Jonathan Berant
Researchers at Tel-Aviv University and AI2 developed a zero-pass, input-independent framework for interpreting all Transformer parameters by projecting them into the embedding space, unifying prior static interpretability theories. This method offers a common ground for model comparison and enables applications like zero-shot knowledge transfer, achieving approximately 70% accuracy on an IMDB sentiment task when stitching models without additional training.
Key insights
- All Transformer parameters can be consistently interpreted as operations or relationships between vocabulary items within the embedding space, providing a unified static view of model mechanics.
- The transpose of the embedding matrix (E^T) is empirically shown to be a more effective approximate right-inverse than the pseudo-inverse for top-k based interpretations of projected parameters.
- The embedding space serves as a viable interface for aligning the internal functionality of different Transformer models and for transferring learned capabilities through 'zero-shot stitching' between models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.