Reproduce “On Initializing Transformers with Pre-trained Embeddings” on OpenResearch

About this paper

Ha Young Kim, Niranjan Balasubramanian, Byungkon Kang

Kim, Balasubramanian, and Kang from SUNY Korea and Stony Brook University investigated why pre-trained embeddings often underperform random initialization in transformer models, revealing that incompatible statistical distributions and interactions with positional encodings are key factors. Their research demonstrates that standardizing pre-trained embeddings to match Xavier distribution significantly improves their utility, often outperforming pure random initialization and preserving semantic information.

Key insights

• Transformer models are sensitive to the statistical distribution of their input embedding parameters, especially their variance.
• Pre-trained embeddings with significantly higher variance than positional encodings can diminish the effect of positional information, disrupting the transformer's ability to model sequential relationships.
• The semantic information in pre-trained embeddings is valuable for transformers, but its benefit is often masked by distribution incompatibilities.

Summary by alphaXiv.