Reproduce “Always Skip Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiping Ji, Hemanth Saratchandran, Peyman Moghadam, Simon Lucey
Researchers at Adelaide University and CSIRO found that the self-attention mechanism in Vision Transformers requires skip connections to prevent catastrophic training failure, attributing this to its intrinsic ill-conditioning. They introduced Token Graying, a pre-conditioning method that improved ViT accuracy on ImageNet-1K by up to 0.3% with minimal computational cost.
Key insights
- The self-attention mechanism is intrinsically ill-conditioned, with its output embedding's condition number bounded by the cube of the input's, making it uniquely reliant on skip connections for stable optimization.
- Skip connections serve as fundamental regularizers for self-attention, critically stabilizing gradient flow and signal propagation, rather than merely offering a performance benefit.
- Pre-conditioning input tokens through methods like DCT Token Graying can further improve ViT performance and conditioning, complementing the role of skip connections.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.