Reproduce “Rethinking Skip Connection with Layer Normalization in Transformers and ResNets” on OpenResearch

About this paper

Fenglin Liu, Xuancheng Ren, Zhiyuan Zhang, Xu Sun, Yuexian Zou

Peking University researchers systematically re-evaluated the skip connection in deep neural networks, demonstrating that expanding the skip connection's scaling factor (λ > 1) can yield superior performance when coupled with Layer Normalization, particularly through their proposed Recursive Skip Connection with Layer Normalization (rSkip+LN). This approach consistently improved models on image classification and machine translation tasks, with gains of up to 1.2 BLEU points and 1.02 absolute error rate reduction on deep models.

Key insights

• Simply expanding the skip connection's scale (λ > 1) without subsequent normalization (`xSkip`) severely degrades performance due to gradient malformation.
• Layer Normalization is highly effective at mitigating gradient instability arising from expanded skip connections, enabling performance gains when `λ > 1` (`xSkip+LN`).
• The recursive application of Layer Normalization within skip connections (`rSkip+LN`) further enhances stability and allows for an adaptive incorporation of the input signal, leading to the best performance.

Summary by alphaXiv.