Reproduce “Rethinking and Improving Multi-task Learning for End-to-end Speech Translation” on OpenResearch

About this paper

Yuhao Zhang, Chen Xu, Bei Li, Hao Chen, Tong Xiao, Chunliang Zhang, Jingbo Zhu

Northeastern University and NiuTrans Research present an improved multi-task learning (MTL) approach for end-to-end speech translation, building on a detailed analysis of task consistency. The method achieves state-of-the-art performance on the MuST-C dataset with significantly reduced training time and eliminates the need for a separate fine-tuning stage.

Key insights

• Gradient consistency between the primary speech translation (ST) task and auxiliary tasks (ASR, MT) varies significantly across different neural network modules (acoustic encoder, textual encoder, decoder) and training stages.
• The textual encoder, which processes both acoustic features and text embeddings, faces unique challenges due to inherent length disparity and representation differences between noisy speech and clean textual information.
• ASR assistance is primarily beneficial in the early training stages and for the acoustic encoder, while its strong influence on the textual encoder can reduce consistency with the MT task.

Summary by alphaXiv.