Reproduce “Adapting Multi-Lingual ASR Models for Handling Multiple Talkers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenda Li, Yao Qian, Zhuo Chen, Naoyuki Kanda, Dongmei Wang, Takuya Yoshioka, Yanmin Qian, Michael Zeng
Researchers from Shanghai Jiao Tong University and Microsoft developed an enhanced Serialized Output Training (SOT) framework with adapter-based fine-tuning to adapt the Whisper ASR model for multi-talker scenarios. This approach enables a single model to accurately transcribe overlapping speech, count speakers, and predict timestamps while maintaining its original multilingual capabilities.
Key insights
- Integrating timestamp prediction directly into the Serialized Output Training (SOT) framework allows the model to learn temporal relationships in overlapping speech.
- Adapter-based transfer learning is effective in adapting large pre-trained ASR models like Whisper for multi-talker scenarios while preserving their original multilingual knowledge.
- A single model can be trained to simultaneously perform multi-talker ASR, count speakers, and predict utterance timestamps.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.