Reproduce “Advancing Multi-talker ASR Performance with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohan Shi, Zengrui Jin, Yaoxun Xu, Yong Xu, Shi-Xiong Zhang, Kun Wei, Yiwen Shao, Chunlei Zhang and 1 more
Researchers at Tencent AI Lab developed an LLM-based Serialized Output Training (SOT) approach for multi-talker Automatic Speech Recognition (ASR) that integrates pre-trained speech encoders with large language models. This method achieved state-of-the-art performance on the real-world AMI meeting corpus, with a 21.0% cpWER, notably outperforming prior methods trained with significantly more supervised data.
Key insights
- Large Language Models' (LLMs) strong long-context awareness is exceptionally well-suited for processing Serialized Output Training (SOT)-style multi-talker transcriptions.
- LLM-based models demonstrate a greater performance advantage in complex, real-world multi-talker conversational scenarios compared to simpler, simulated datasets.
- A well-adapted speech encoder combined with multi-stage training and LoRA fine-tuning of the LLM is crucial for optimal performance in this architecture.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.