Reproduce “Probing the Emergence of Cross-lingual Alignment during LLM Training” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Hetong Wang, Pasquale Minervini, Edoardo M. Ponti

Researchers at the University of Edinburgh investigated how cross-lingual alignment emerges in BLOOM Large Language Models during their pre-training, establishing a strong correlation between shared neural subnetworks and zero-shot cross-lingual transfer performance. The study found that this alignment does not always grow monotonically, observing periods of degradation in smaller models that are mitigated by increased model scale.

Key insights

  • A direct and robust correlation exists between the degree of implicit cross-lingual alignment (neuron overlap) and an LLM's ability to perform zero-shot cross-lingual transfer on downstream tasks.
  • The emergence of cross-lingual alignment during pre-training is non-monotonic, with smaller models experiencing dramatic drops in alignment and performance at specific training stages.
  • Model scale plays a crucial role in the stability of alignment emergence, with larger models exhibiting more consistent and monotonic growth in cross-lingual capabilities, suggesting a scaling law for this phenomenon.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.