Reproduce “Multilingual Large Language Models and Curse of Multilinguality” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daniil Gurgurov, Tanja Bäumel, Tatiana Anikina

This survey paper provides an introductory overview of multilingual large language models (LLMs), detailing their architectural foundations, pre-training objectives, and data strategies. It highlights the "curse of multilinguality" as a central challenge, where increasing language coverage can diminish per-language performance, and reviews current research approaches to mitigate this limitation.

Key insights

  • Multilingual LLMs primarily rely on the Transformer architecture, adapted into encoder-only, decoder-only, and encoder-decoder configurations for different understanding and generation tasks.
  • Effective cross-lingual processing is facilitated by a combination of diverse pre-training objectives and sophisticated subword tokenization strategies that handle linguistic variety and rare words.
  • The "curse of multilinguality" is a fundamental limitation caused by linguistic diversity, data sparsity, and model complexity, hindering uniformly high performance across all supported languages.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.