Reproduce “Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhenyu Guo, Wenguang Chen
A modular Transformer architecture separates knowledge from reasoning, utilizing a generalized cross-attention mechanism to query a shared knowledge base. The work mathematically demonstrates that the standard Transformer's Feed-Forward Network (FFN) functions as a specialized form of this generalized cross-attention, effectively re-interpreting FFNs as implicit knowledge retrieval modules.
Key insights
- The Feed-Forward Network (FFN) in a standard Transformer is mathematically derived to be a specialized instance of the proposed generalized cross-attention mechanism.
- FFNs implicitly perform knowledge retrieval from an internal, compressed 'Implicit E' (knowledge base) learned during pre-training, validating the FFN-as-key-value-memory hypothesis.
- The generalized cross-attention mechanism uses sparse activation and knowledge-specific thresholds to enable more selective and context-dependent knowledge access than standard attention.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.