Reproduce “Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lingwei Meng, Jiawen Kang, Yuejiao Wang, Zengrui Jin, Xixin Wu, Xunying Liu, Helen Meng
Researchers at The Chinese University of Hong Kong developed a parameter-efficient framework that empowers the frozen Whisper model to perform joint multi-talker and target-talker speech recognition. The system achieves new state-of-the-art results on LibriMix datasets for both tasks, demonstrating high efficiency and retaining multilingual capabilities.
Key insights
- The framework achieves high performance in joint multi-talker and target-talker ASR while being highly parameter-efficient, with only 1-3% of the total model parameters being trainable.
- Training with the Target Talker Identifier (TTI) module unexpectedly improves general multi-talker ASR performance, suggesting that learning to differentiate speakers for the target task enhances overall speaker disentanglement.
- The adapted Whisper model largely retains its inherent multilingual capabilities, showing acceptable zero-shot performance and rapid adaptation to new languages like Mandarin.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.