Reproduce “OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment” on OpenResearch

About this paper

Xize Cheng, Tao Jin, Linjun Li, Wang Lin, Xinyu Duan, Zhou Zhao

The OpenSR framework enables zero-shot lip-reading by leveraging an audio-trained decoder combined with a frozen, pre-aligned multi-modal encoder, thereby overcoming visual data scarcity. It establishes new state-of-the-art performance on LRS2 with a 25.0% Word Error Rate (WER) for lip-reading and 2.7% WER for audio-visual speech recognition.

Key insights

• Maintaining the multi-modality alignment learned during pre-training, by freezing the encoder, is crucial for enabling zero-shot modality transfer from audio to visual input.
• Linguistic knowledge and domain-specific word distributions can be effectively learned by a decoder from labeled audio data and then applied to visual input through a shared, pre-aligned phoneme space.
• The Cluster Prompt strategy efficiently improves recognition of visually similar words (homophenes) in few-shot visual learning settings by providing fine-grained, cluster-specific biases.

Summary by alphaXiv.