Reproduce “Grouped Differential Attention” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junghwan Lim, Sungmin Lee, Dongseok Kim, Wai Ting Cheung, Beomgyu Kim, Taehwan Kim, Haesol Lee, Junhyeok Lee and 2 more

Motif Technologies developed Grouped Differential Attention (GDA), an extension of Differential Attention that uses an unbalanced allocation of attention heads for signal preservation and noise control, alongside a group-differentiated growth strategy for scaling. Experiments demonstrated that a 3:1 ratio yielded an average gain of +0.88% over the 1:1 baseline in pretraining, and a 4:1 ratio achieved a +2.54% average gain during progressive continual training, including a +47.17% increase on GSM8K.

Key insights

  • An unbalanced, moderate allocation ratio between signal-preserving and noise-control heads can enhance model generalization and performance within fixed computational budgets.
  • The assumption of symmetric head allocation in Differential Attention is inefficient; strategic resource distribution is more effective.
  • Selective, group-differentiated growth, focusing on expanding signal-processing components, provides a more efficient and stable pathway for scaling Transformer models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.