Reproduce “Peri-LN: Revisiting Normalization Layer in the Transformer Architecture” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jeonghoon Kim, Byeongchan Lee, Cheonbok Park, Yeontaek Oh, Beomjun Kim, Taehwan Yoo, Seongjin Shin, Dongyoon Han and 2 more
Researchers from NAVER Cloud and KAIST rigorously analyze Peri-LN, a Transformer normalization strategy that normalizes both sub-layer inputs and outputs, demonstrating its capability to maintain stable hidden-state activations and gradient flow, resulting in improved pre-training and downstream task performance for large language models.
Key insights
- Peri-LN effectively constrains the growth of activation variance to a linear or sub-exponential trajectory across layers and training iterations, preventing the 'massive activations' seen in Pre-LN.
- The output normalization component of Peri-LN acts as a self-regularizing mechanism during backward propagation, damping gradient norms and preventing explosions that commonly destabilize Pre-LN training.
- Peri-LN exhibits enhanced robustness to hyperparameters and is compatible with FP16 precision because its controlled hidden-state activations consistently remain within the representable bounds, unlike Pre-LN which often exceeds them.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.