Reproduce “Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM” on OpenResearch

About this paper

Alexander Podolskiy, Semen Molokov, Timofey Gerasin, Maksim Titov, Alexey Rukhovich, Artem Khrapov, Kirill Morozov, Evgeny Tetin and 9 more

The Gamayun team developed a 1.5-billion-parameter multilingual language model, Gamayun, trained from scratch on 2.5 trillion tokens using a novel two-stage pre-training strategy. This model achieved competitive performance against larger baselines and state-of-the-art results on Russian-specific benchmarks, demonstrating effective multilingual capabilities with a constrained computational budget.

Key insights

• Initial naïve language mixing degrades performance in primary languages, validating the 'curse of multilinguality,' but this can be effectively mitigated through a dynamic, multi-stage data mixing strategy.
• Dynamically increasing the proportion of high-quality English data after establishing cross-lingual alignment effectively transfers knowledge and improves overall multilingual performance.
• Training a multilingual model from scratch on relevant web data allows for the acquisition and preservation of deep, culturally specific knowledge in smaller models, as demonstrated with Russian cultural benchmarks.

Summary by alphaXiv.