Reproduce “AISHELL6-whisper: A Chinese Mandarin Audio-visual Whisper Speech Dataset with Speech Recognition Baselines” on OpenResearch

About this paper

Cancan Li, Fei Su, Juan Liu, Hui Bu, Yulong Wan, Hongbin Suo, Ming Li

Researchers from Wuhan University, Duke Kunshan University, Beijing AISHELL Technology, and OPPO's AI Center released AISHELL6-Whisper, the largest audio-visual parallel whisper speech dataset for Chinese Mandarin, with 30 hours each of whisper and normal speech. Their proposed audio-visual speech recognition (AVSR) baseline, based on Whisper-Flamingo and enhanced with parallel training and a whisper-specific projection layer, achieved a 4.13% Character Error Rate (CER) for whisper speech on the new dataset and established new state-of-the-art performance on the English wTIMIT benchmark.

Key insights

• Creating large-scale, high-quality audio-visual datasets is crucial for bridging the performance gap between normal and whisper speech recognition.
• A parallel training approach, leveraging the relationship between whisper and normal speech, combined with a whisper-specific projection layer, effectively addresses the acoustic challenges of whisper speech.
• Multimodal fusion, specifically the integration of visual cues like lip movements, significantly enhances the accuracy and robustness of whisper speech recognition, especially in challenging acoustic conditions.

Summary by alphaXiv.