Reproduce “Scheduled Interleaved Speech-Text Training for Speech-to-Speech Translation with LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hayato Futami, Emiru Tsunoo, Yosuke Kashiwagi, Yuki Ito, Hassan Shahmohammadi, Siddhant Arora, Shinji Watanabe
Sony Group Corporation and Carnegie Mellon University researchers develop Scheduled Interleaved Speech-Text Training (SILT), a progressive fine-tuning strategy that enables text-based LLMs to perform speech-to-speech translation by gradually transitioning from text-dominant to speech-only training through word-level aligned interleaving of discrete speech units and text tokens, achieving consistent improvements across seven language pairs with particularly strong gains for low-resource languages (Italian-English improved from 12.8 to 19.5 ASR-BLEU, Russian-English from 6.0 to 14.1, Portuguese-English from 10.3 to 19.5) while demonstrating that the scheduling component is crucial as constant-ratio interleaving performs worse than baseline, and showing through representation analysis that the method successfully bridges the modality gap between speech and text by leveraging LLaMA3.2-1B's linguistic capabilities while adapting to the longer sequence lengths and different statistical properties of discrete speech units derived from ASR-fine-tuned w2v-BERT encoders.
Key insights
- Gradual modality adaptation is more effective than constant-ratio mixing or immediate speech-only training, as evidenced by superior performance of scheduled versus non-scheduled interleaving
- Both input and output side interleaving is necessary for optimal performance, with single-side interleaving showing substantial performance drops
- The method specifically improves speech-text representation similarity while preserving text-text similarity, indicating targeted modality adaptation rather than general degradation
- Word-level alignment accuracy is crucial for effective interleaving, with misaligned interleaving significantly reducing translation quality
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.