Reproduce “Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models” on OpenResearch

About this paper

Terra Blevins, Tomasz Limisiewicz, Suchin Gururangan, Margaret Li, Hila Gonen, Noah A. Smith, Luke Zettlemoyer

Researchers at the University of Washington and Charles University in Prague introduced Cross-lingual Expert Language Models (X-ELM) to alleviate the "curse of multilinguality" in large language models. This sparse modeling approach, which specializes individual experts using linguistic typology, demonstrably improves per-language performance, particularly for low-resource languages, and facilitates robust adaptation to new languages without compromising existing knowledge.

Key insights

• Leveraging linguistic typology for data allocation to specialized experts yields superior performance compared to arbitrary or TF-IDF-based clustering, highlighting the benefit of domain-specific knowledge.
• Targeted multilinguality, where experts cover small, typologically related language groups, can be more effective than strictly monolingual experts or dense models, suggesting an optimal level of cross-lingual sharing.
• The sparse expert training paradigm enables significant computational efficiencies by allowing independent expert training, reducing synchronized GPU requirements and potentially democratizing multilingual NLP development.

Summary by alphaXiv.