Reproduce “Multitask Learning with Low-Level Auxiliary Tasks for Encoder-Decoder Based Speech Recognition” on OpenResearch

About this paper

Shubham Toshniwal, Hao Tang, Liang Lu, Karen Livescu

This paper from Toyota Technological Institute at Chicago presents a multitask learning approach that integrates low-level auxiliary tasks into an attention-based encoder-decoder model for speech recognition. By applying phoneme-based or HMM state-level supervision at intermediate encoder layers, the model achieves improved recognition accuracy, reducing Word Error Rate on the Switchboard/CallHome Eval2000 test set from 33.7% to 32.0% compared to a strong baseline.

Key insights

• Applying low-level auxiliary losses (e.g., phoneme decoding, HMM state classification) at corresponding lower layers of a deep encoder-decoder model is more effective than at higher layers for improving ASR accuracy.
• Different types of low-level supervision (phoneme-based and state-level) can be combined synergistically to further enhance model performance.
• Multitask learning with intermediate supervision not only aids optimization but also contributes to better generalization, suggesting the model learns more robust intermediate representations.

Summary by alphaXiv.