Reproduce “Tracing the Evolution of Word Embedding Techniques in Natural Language Processing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minh Anh Nguyen, Kuheli Sai, Minh Nguyen
A historical and bibliometric study maps the development of NLP representations from distributional and count-based methods to static, contextual, and sentence-level embeddings across 149 papers published between 1954 and 2025. Comparing 95 pre-GPT-3 papers with 54 later studies, it finds a marked shift toward contextual and sentence/document representations, larger research teams, and longer abstracts, alongside growing involvement from major technology companies.
Key insights
- Modern contextual and generative representations extend earlier ideas from distributional semantics, corpus statistics, dimensionality reduction, and geometric representation rather than replacing them outright.
- After GPT-3, research emphasis moved away from static and statistical embeddings toward context-dependent token representations and sentence/document encoders; only 10 of the identified techniques appeared in both periods.
- The expansion of large-scale modeling coincided with larger teams and stronger participation from major technology companies, raising concerns about computational concentration, research access, and coverage of low-resource languages.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.