Reproduce “Reproducing and Dissecting Denoising Language Models for Speech Recognition” on OpenResearch

About this paper

Dorian Koch, Albert Zeyer, Nick Rossenbach, Ralf Schlüter, Hermann Ney

Researchers from RWTH Aachen University and AppTek conducted the first independent, large-scale empirical study of Denoising Language Models (DLMs) for speech recognition, releasing a reproducible open-source pipeline. Their work demonstrates that DLMs surpass traditional language models in accuracy once a certain computational budget is met, especially when using the newly proposed DLM-sum decoding method, achieving state-of-the-art results on LibriSpeech and Loquacious datasets.

Key insights

• DLMs can outperform traditional autoregressive LMs for ASR error correction but require a 'compute tipping point' of sufficient computational resources and training time to demonstrate their superior scaling and efficiency.
• Integrating richer information from the ASR's hypothesis space, such as through the novel DLM-sum decoding method or by conditioning the DLM encoder on dense k-probability inputs, significantly enhances DLM robustness and performance.
• A carefully designed data generation pipeline, including the TTS-ASR process and a variety of data augmentation techniques (e.g., SpecAugment, dropout, mixup), is crucial for training effective DLMs.

Summary by alphaXiv.