Reproduce “Improving Transformers with Dynamically Composable Multi-Head Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Da Xiao, Qingye Meng, Shengping Li, Xingyuan Yuan
Dynamically Composable Multi-Head Attention (DCMHA) enhances Transformer efficiency and expressive power by enabling attention heads to dynamically compose in an input-dependent manner, serving as a drop-in replacement for standard Multi-Head Attention. This architectural improvement allows a 6.9B parameter model to outperform a 12B baseline, demonstrating a 1.7x reduction in parameters for equivalent or better performance.
Key insights
- Dynamic composition is substantially more effective than static methods for head interaction, validating the importance of input-dependent adaptability.
- Composing attention weights post-softmax is more impactful for performance gains than composing attention scores pre-softmax.
- Low-rank dynamic projection branches contribute more to performance improvement than dynamic gating branches, with an optimal projection rank of 2.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.