Reproduce “Advanced Modeling of Interlanguage Speech Intelligibility Benefit with L1-L2 Multi-Task Learning Using Differentiable K-Means for Accent-Robust Discrete Token-Based ASR” on OpenResearch

About this paper

Kentaro Onda, Satoru Fukayama, Daisuke Saito, Nobuaki Minematsu

An advanced computational model of Interlanguage Speech Intelligibility Benefit (ISIB) improves Automatic Speech Recognition (ASR) robustness to foreign-accented speech by integrating differentiable k-means with multi-task learning for L1 and L2 ASR. It achieved approximately a 20% relative reduction in Word Error Rate (WER) for Japanese-accented English when adapting with limited accented data.

Key insights

• An L1-initialized discrete phonetic token set, when fine-tuned via multi-task learning for both L1 and L2 ASR, can effectively model the Interlanguage Speech Intelligibility Benefit (ISIB), mirroring how human non-native listeners perceive speech.
• Jointly optimizing the self-supervised learning (SSL) model and k-means centroids in an end-to-end differentiable manner leads to more robust and phonetically meaningful tokens for accent-robust ASR.
• The weighting of L1 and L2 ASR tasks during multi-task learning allows for tailoring the token space to either native or accented speech recognition, with an L1 bias improving accented speech perception.

Summary by alphaXiv.