Reproduce “Fixup Initialization: Residual Learning Without Normalization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongyi Zhang, Yann N. Dauphin, Tengyu Ma
Fixup Initialization enables stable training of extremely deep residual networks and Transformer models without normalization layers, matching or surpassing the performance of models using Batch Normalization or Layer Normalization when combined with appropriate regularization. The work demonstrates that careful initialization can control gradient dynamics and signal propagation, dispelling the long-held belief that normalization is indispensable for deep network stability.
Key insights
- The exploding gradient problem in deep residual networks without normalization is primarily due to uncontrollably large logits at initialization, as identified through theoretical analysis.
- Fixup Initialization demonstrates that the perceived benefits of normalization—stable training, high learning rates, fast convergence, and good generalization—are not unique to normalization layers but can be achieved through a carefully designed static initialization.
- Normalization layers provide an implicit regularization effect that Fixup-initialized models may need to compensate for externally with explicit regularization techniques like Mixup or Cutout to prevent overfitting.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.