Reproduce “Multilingual Definition Modeling” on OpenResearch

About this paper

Edison Marrese-Taylor, Erica K. Shimomoto, Alfredo Solano, Enrique Reid

Researchers introduce the first truly multilingual study on definition modeling by collecting new, non-crowdsourced dictionary datasets for Spanish, French, Portuguese, and German. Their work empirically evaluates multilingual language models and Large Language Models on monosemic definition generation, demonstrating LLMs generally outperform fine-tuned models but exhibit challenges with output language control. The study also establishes definition modeling as an efficient, natural, and compute-constrained alternative for evaluating multilingual LLM proficiency.

Key insights

• Multilingual fine-tuning of mT5 for monosemic definition generation did not consistently demonstrate cross-lingual synergies, performing comparably to monolingual fine-tuning.
• Large Language Models generally achieved superior definition generation quality but frequently failed to adhere to explicit instructions for output language and showed limited, inconsistent efficacy from in-context learning.
• Definition modeling performance, specifically via BERTScore, strongly correlates with scores on established multilingual benchmarks (HellaSwag, ARC-Challenge, MMLU), positioning it as an efficient and natural LLM evaluation method.

Summary by alphaXiv.