Reproduce “Decoding while Adapting: Zero-Shot Online Speaker Adaptation via Audio-Textual Prompts for Elderly Speech Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chengxi Deng, Xurong Xie, Shujie Hu, Mengzhe Geng, Tianzi Wang, Youjun Chen, Huimeng Wang, Haoning Xu and 2 more
Researchers from The Chinese University of Hong Kong developed an online, zero-shot speaker adaptation framework for elderly speech recognition, leveraging cross-utterance audio-textual prompts to enhance a fine-tuned Whisper model. The method achieved a 0.61% absolute WER reduction on English DementiaBank Pitt and a 1.22% absolute CER reduction on Cantonese JCCOCC MoCA, while providing up to 9.83 times faster inference compared to batch-mode adaptation.
Key insights
- Integrating both history speech and decoded textual information (audio-textual prompts) provides a more comprehensive and accurate speaker representation, jointly addressing acoustic and language deficiencies, compared to relying solely on audio features.
- The Q-Former module effectively condenses variable-length multimodal contextual information into compact speaker prompts, enabling efficient online adaptation without the latency of batch processing.
- Dual Cross-Modality Fusion facilitates superior interaction between history speech and text, proving more effective for multimodal speaker modeling than other fusion strategies.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.