Reproduce “Word Definitions from Large Language Models” on OpenResearch

About this paper

Bach Pham, JuiHsuan Wong, Samuel Kim, Yunting Yin, Steven Skiena

Researchers systematically compared word definitions generated by large language models (LLMs) with those from traditional dictionaries and word embeddings, finding that GPT-4 definitions exhibit high accuracy and semantic consistency across word frequencies, aligning closely with established lexicographical sources. The study further indicates that definition-based embeddings, including those from LLMs, may offer more reliable semantic representations for low-frequency words than traditional co-occurrence methods.

Key insights

• LLMs primarily synthesize definitions rather than directly repeating classical dictionary entries, exhibiting less surface-level textual similarity than inter-dictionary comparisons.
• GPT-4 definitions demonstrate high accuracy and strong semantic consistency with traditional dictionary definitions across various parts of speech and word frequency tiers.
• Traditional word embeddings like GloVe and FastText show decreased accuracy and consistency for low-frequency words, indicating a limitation in capturing semantics for rare terms.

Summary by alphaXiv.