Reproduce “Differential Transformer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei
DIFF Transformer replaces conventional attention with a difference of two softmax query–key maps, one scaled by a learned scalar and subtracted from the other before weighting values. In a controlled 4K-token test, 3B models retrieved two queried city–number facts among six with 85% accuracy versus 55% for Transformer. This gain was measured in distractor-heavy retrieval, not as universal long-context robustness.
Key insights
- The 'differential attention' mechanism actively cancels attention noise by computing the difference between two softmax attention maps, akin to noise-canceling technology.
- Applying headwise GroupNorm is crucial for stabilizing training in the Differential Transformer due to the sparser attention patterns produced by the differential mechanism.
- The architecture inherently reduces activation outliers, a property that facilitates more aggressive and effective model quantization compared to standard Transformers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.