Reproduce “Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shaoxiong Ji, Zihao Li, Jaakko Paavola, Hengyu Luo, Jörg Tiedemann
University of Helsinki researchers demonstrated that including bilingual translation data during continual pre-training of Llama 3 and Llama 3.1 models to 500 languages significantly boosts performance for low-resource languages, notably achieving up to 140% BLEU/chrF++ gains in machine translation. They also released a new MaLA bilingual translation corpus and the EMMA-500 models.
Key insights
- Continual pre-training with bilingual translation data significantly enhances multilingual performance and language transfer, particularly benefiting low-resource languages, compared to monolingual-only adaptation.
- Highly pre-trained base models like Llama 3 and Llama 3.1 demonstrate increased resistance to further multilingual adaptation across hundreds of additional languages compared to earlier generations like Llama 2.
- Massively multilingual adaptation can introduce performance trade-offs, with gains in low-resource languages sometimes accompanied by minor performance degradation in high-resource languages.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.