Reproduce “Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zixuan Jiang, Jiaqi Gu, Hanqing Zhu, David Z. Pan
This work formally establishes the arithmetic equivalence of pre-normalization LayerNorm, RMSNorm, and a new Compressed RMSNorm (CRMSNorm) in Transformer architectures. By systematically eliminating redundant mean information, the proposed methods offer computationally more efficient alternatives, achieving up to a 10% reduction in inference time for large models without requiring retraining or performance degradation.
Key insights
- LayerNorm and RMSNorm are arithmetically equivalent when applied to zero-mean input vectors, allowing for direct substitution under specific architectural modifications.
- The mean information processed by LayerNorm in pre-normalization Transformers is redundant and can be systematically removed or compressed without altering the model's functionality.
- Lossless compression of zero-mean vectors is possible, enabling further efficiency improvements by operating on a smaller, `(d-1)`-dimensional vector space.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.