Reproduce “Streaming Speech-to-Text Translation with a SpeechLLM” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Titouan Parcollet, Shucong Zhang, Xianrui Zheng, Rogier C. van Dalen
Researchers from Samsung AI Center – Cambridge introduce an "intermixed" SpeechLLM architecture that integrates a learned, adaptive wait policy for real-time speech-to-text translation. This system achieves competitive translation quality at 1-2 second latency and demonstrates robustness to varying acoustic inputs, significantly outperforming fixed-policy streaming baselines in terms of both quality and latency.
Key insights
- Directly integrating an adaptive "Wait" token into the LLM's generative process allows the model to learn and execute a dynamic, context-aware streaming policy for speech-to-text translation.
- Decoupling early "wait" decisions via a specialized, energy-efficient head can significantly reduce the computational load for streaming LLMs without degrading the final translation quality.
- Generating training supervision using robust phrase-level speech-to-text alignments is essential for effectively teaching streaming models when to emit translation tokens, particularly across linguistically diverse language pairs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.