Reproduce “The Geometry of Multilingual Language Models: An Equality Lens” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Cheril Shah, Yashashree Chandak, Manan Suri

This research quantifies the geometric properties of language representations within mBERT, MiniLM, and XLM-RoBERTa using new metrics, the Cross-Lingual Similarity Index and Geometric Separability Index. The analysis reveals representational inequalities across languages, with low-resource languages consistently exhibiting lower cross-lingual similarity and higher geometric separability compared to high-resource languages.

Key insights

  • Multilingual models do not represent all languages equally; low-resource languages consistently show poorer quality representations in terms of cross-lingual similarity.
  • Languages tend to form largely separable vector spaces rather than converging into a unified "interlingua" in mBERT and XLM-RoBERTa, which may hinder optimal cross-lingual transfer.
  • The geometric properties of language representations are influenced by model architecture, pre-training objectives, and training data composition, affecting how languages are embedded and relate to each other.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.