Reproduce “Conformer-based Target-Speaker Automatic Speech Recognition for Single-Channel Audio” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yang Zhang, Krishna C. Puvvada, Vitaly Lavrukhin, Boris Ginsburg
NVIDIA researchers developed CONF-TSASR, a Conformer-based Target-Speaker Automatic Speech Recognition (TS-ASR) system that extracts and transcribes the speech of a specific individual from multi-speaker audio mixtures using a reference voice sample. The system achieved a 4.2% Target-Speaker Word Error Rate (TS-WER) on WSJ0-2mix-extr, setting new performance benchmarks across multiple datasets including WSJ0-3mix-extr, LibriSpeech2Mix, and LibriSpeech3Mix.
Key insights
- Integrating a scale-invariant spectrogram reconstruction loss during training significantly improves the model's ability to isolate the target speaker's speech.
- The model's performance on multi-speaker audio is significantly affected by the Signal-to-Noise Ratio (SNR) and the length of the auxiliary utterance, with higher SNRs and longer utterances leading to better accuracy.
- Conformer-based networks are effective for both masking (MaskNet) and transcription (ASR Module) in Target-Speaker ASR, leveraging their ability to capture both local and global context.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.