Reproduce “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale” on OpenResearch

About this paper

Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen and 5 more

Meta AI's XLS-R introduces a self-supervised cross-lingual speech representation model trained on 436,000 hours of audio across 128 languages, achieving new state-of-the-art results in speech translation (e.g., +7.4 BLEU for X→English), recognition, and classification tasks. The 2-billion-parameter model significantly benefits low-resource languages and performs competitively with monolingual models at scale.

Key insights

• Scaling self-supervised cross-lingual speech pre-training in terms of data volume, language coverage, and model size consistently leads to substantial performance improvements across various speech tasks, particularly for low-resource languages.
• Increased model capacity (up to 2 billion parameters) enables cross-lingual models to overcome 'capacity dilution' and perform competitively with, or even surpass, strong monolingual models on high-resource languages.
• Multilingual fine-tuning and initializing the decoder with pre-trained text-to-text models like mBART are effective strategies for enhancing performance, especially in low-supervision speech translation scenarios.

Summary by alphaXiv.