Reproduce “Unified Streaming and Non-streaming Two-pass End-to-end Model for Speech Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Binbin Zhang, Di Wu, Zhuoyuan Yao, Xiong Wang, Fan Yu, Chao Yang, Liyong Guo, Yaguang Hu and 2 more
The U2 model unifies streaming and non-streaming automatic speech recognition within a single hybrid CTC/attention architecture, employing a two-pass decoding strategy. This approach achieves 4.64% Character Error Rate (CER) on AISHELL-1 for non-streaming and 5.33% CER for streaming at 320ms latency, outperforming prior dedicated streaming solutions.
Key insights
- A two-pass strategy, leveraging the speed of streaming CTC for hypothesis generation and the accuracy of full-context attention for rescoring, effectively bridges the performance gap between streaming and non-streaming ASR.
- Dynamic chunk training allows a single model to learn and generalize across a wide spectrum of context lengths, from real-time streaming to offline processing, eliminating the need for multiple specialized models.
- Causal convolutions within the Conformer encoder are essential for precisely controlling the right context, enabling a single chunk size parameter to dynamically trade off latency and accuracy at inference time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.