Reproduce “Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR” on OpenResearch

About this paper

Dongseong Hwang, Khe Chai Sim, Yu Zhang, Trevor Strohman

Researchers at Google LLC systematically compared soft and hard target distillation for large-scale Recurrent Neural Network Transducers (RNN-T) in Automatic Speech Recognition (ASR). Their work demonstrates that soft target distillation achieves new state-of-the-art Word Error Rates on LibriSpeech for large models with similar teacher-student architectures and effectively enables continuous adaptation of production ASR models to unlabeled data domains.

Key insights

• Soft target distillation is superior when the teacher and student RNN-T models have similar or identical architectures, particularly in self-training scenarios.
• Hard target distillation performs better when there is a significant architectural mismatch between the teacher and student models, such as distilling from a large bi-directional model to a smaller streaming model.
• Soft target distillation, combined with novel frequency augmentations (FreqAug), enables effective continuous adaptation of large-scale production ASR models to new, unlabeled data domains without performance degradation.

Summary by alphaXiv.