Reproduce “MaLA-500: Massive Language Adaptation of Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André F. T. Martins, Hinrich Schütze
MaLA-500, an adaptation of the LLaMA 2-7B model, extends large language model capabilities to 534 languages, addressing the performance gap for low-resource languages. The model achieved a negative log-likelihood of 151.25 on the Glot500-c test set and macro-average accuracies of 57.02% on SIB200 and 48.89% on Taxi1500, outperforming comparable open LLMs.
Key insights
- Massive language adaptation via vocabulary extension and parameter-efficient continued pretraining (LoRA) can successfully scale generative LLMs like LLaMA 2 to hundreds of diverse languages.
- The effectiveness of vocabulary extension, measured by segmentation length reduction, shows a moderate to high correlation (0.53 Pearson) with performance gains on downstream tasks, particularly for languages with non-Latin scripts.
- In-context learning capabilities of the adapted model improve significantly with an increased number of shots, demonstrating its ability to learn from few examples across a massively multilingual context.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.