Reproduce “Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mor Geva, Avi Caciularu, Kevin Ro Wang, Yoav Goldberg
Feed-Forward Network (FFN) layers in Transformer-based Language Models build predictions by dynamically promoting human-interpretable concepts within the vocabulary space, rather than primarily eliminating tokens. Decomposing FFN outputs into individual 'sub-updates' provides a mechanistic understanding that enables practical applications in controlling generation and improving computational efficiency.
Key insights
- Individual value vectors within FFN layers encode human-interpretable concepts (semantic or syntactic categories) that are consistently found across all layers of the Transformer.
- FFN updates primarily operate through a 'promotion mechanism,' assigning substantially higher positive scores to tokens that are successfully chosen as the top candidate, while 'elimination' is often a passive process.
- The dynamics of dominant FFN sub-updates provide a strong signal for prediction stability, particularly in later layers where some functional value vectors contribute to both promoting and actively suppressing tokens.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.