Reproduce “Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Srijith Radhakrishnan, Chao-Han Huck Yang, Sumeer Ahmad Khan, Rohit Kumar, Narsis A. Kiani, David Gomez-Cabrero, Jesper N. Tegner
King Abdullah University of Science and Technology (KAUST) researchers developed Whispering LLaMA, a cross-modal framework that performs generative error correction for Automatic Speech Recognition (ASR) by integrating acoustic features from Whisper into a frozen LLaMA model using parameter-efficient adapters. This approach achieved a 37.66% relative Word Error Rate improvement over the n-best Oracle and demonstrated effective utilization of acoustic information.
Key insights
- The proposed cross-modal fusion mechanism, particularly the specific weight initialization and gated fusion, is critical for stable training and effective utilization of acoustic features from Whisper.
- Masking the prompt during loss calculation is crucial for the model to learn accurate semantics for error correction, avoiding penalties for errors in initial hypotheses.
- Integrating acoustic information via the Whisper adapter significantly outperforms text-only baselines, confirming the value of cross-modal fusion for ASR error correction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.