Reproduce “On the Dimensionality of Word Embedding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zi Yin, Yuanyuan Shen
Develops a mathematical framework to determine the optimal dimensionality for word embeddings, introducing the Pairwise Inner Product (PIP) loss to analyze the inherent bias-variance tradeoff. The framework shows that algorithms like Word2Vec and GloVe (α=0.5) are robust to overfitting at high dimensions, and its predicted optimal dimensionality aligns with empirical observations.
Key insights
- The optimal dimensionality for word embeddings arises from a bias-variance tradeoff in the Pairwise Inner Product (PIP) loss, where lower dimensions increase bias and higher dimensions increase variance.
- Word embedding algorithms characterized by α = 0.5, such as Word2Vec and GloVe, exhibit greater robustness to overfitting at higher dimensions compared to algorithms with different α values.
- The PIP loss framework provides a mathematical explanation for the commonly observed empirical pattern where word embedding performance rapidly improves with increasing dimensionality up to a point, then plateaus or decreases.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.