Reproduce “Zipfian Whitening” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sho Yokoi, Han Bao, Hiroto Kurita, Hidetoshi Shimodaira
A new post-processing technique for word embeddings, Zipfian Whitening, incorporates empirical word frequencies to address the anisotropy problem and improves embedding quality, outperforming standard methods. The approach also provides a theoretical framework explaining the success of several existing natural language processing models.
Key insights
- Establishes a theoretical framework using exponential families, showing that informative words (those that significantly alter word distributions) naturally receive longer vector norms under a Zipfian prior.
- Demonstrates that the success of existing methods like WhiteningBERT and Skip-Gram with Negative Sampling can be attributed to their implicit incorporation of Zipfian frequency information.
- Introduces frequency-aware evaluation metrics (Degree of Centrality and Degree of Isotropy) that show strong correlation with downstream task performance, unlike uniform-based metrics, which show no correlation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.