Reproduce “Highway Transformer: Self-Gating Enhanced Self-Attentive Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yekun Chai, Shuo Jin, Xinwen Hou
Researchers from the Institute of Automation, Chinese Academy of Sciences (CASIA) and the University of Pittsburgh developed the Highway Transformer, which integrates Self-Dependency Units (SDU) into standard Transformer blocks. This modification significantly accelerates model convergence during training and improves performance on language modeling tasks, demonstrating up to an 8.76% perplexity reduction on char-level Penn TreeBank.
Key insights
- Self-gating mechanisms, like the proposed Self-Dependency Units (SDUs), can accelerate the convergence of Transformer models and enhance performance by refining feature-wise information.
- SDUs are most beneficial when integrated into shallow Transformer layers, suggesting that these layers primarily learn local context, which benefits more from feature-wise importance.
- Gating mechanisms are applicable and effective in attention-based architectures, broadening their utility beyond recurrent or convolutional networks to control information flow.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.