Reproduce “Turning Speech Language Models into Multilingual Listeners” on OpenResearch

About this paper

Tolúlopé Ògúnrèmí, Dan Jurafsky, Chris Manning, Ahmet Üstün, Martijn Bartelds

MULTISPEECHQA and MULTISPEECH-BENCH provide a scalable route to multilingual spoken instruction following by translating English QA data into 23 languages and synthesizing roughly 9,200 hours of speech. Fine-tuning Qwen2.5-Omni on the resulting data improved spoken QA, winning 60.6% of comparisons against the original model, although gains did not extend to ASR or speech translation and synthetic-data quality varied substantially across languages.

Key insights

• Synthetic translation and speech synthesis can provide useful multilingual spoken instruction data at a scale that is impractical with fully human-recorded examples, but benchmark translation review still required edits for 72% of sampled items.
• The strong cascading ASR-plus-LLM baseline generally beats untuned open-weight SLMs on spoken QA and ASR, showing that end-to-end integration does not automatically provide an advantage.
• Training on all 23 languages outperformed a ten-language mixture, while adding a relatively small amount of speech-translation data produced no consistent benefit.

Summary by alphaXiv.