Reproduce “Delta Attention Residuals” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cheng Luo, Zefan Cai, Junjie Hu
Delta Attention Residuals (DAR) enhance deep transformer architectures by addressing 'routing collapse' in existing Attention Residuals, which used redundant cumulative hidden states, replacing them with a more informative delta-based routing. This method achieved up to an 8.2% perplexity improvement on a 7.57B parameter model, maintained sharper routing, and enabled disruption-free fine-tuning of pretrained models, offering a practical balance of performance and efficiency.
Key insights
- Cumulative hidden states cause 'routing collapse' in deep networks because their increasing redundancy reduces the contrast needed for effective selective routing.
- Routing over 'deltas' (the specific changes introduced by each sublayer) provides naturally diverse and informative signals, maintaining sharp, selective cross-layer communication throughout network depth.
- An additive routing formulation with zero-initialized routing queries allows for 'safe initialization,' preventing performance degradation when fine-tuning pretrained models with new architectural components.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.