Reproduce “Speech Recognition Rescoring with Large Speech-Text Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Prashanth Gurunath Shivakumar, Jari Kolehmainen, Aditya Gourav, Yi Gu, Ankur Gandhe, Ariya Rastrow, Ivan Bulyko
Researchers from Amazon Science developed a rescoring method for Automatic Speech Recognition (ASR) systems using Speech-Text Foundation Models, which are multi-modal large language models trained on both speech and text data. This approach consistently yields greater Word Error Rate (WER) reductions compared to traditional text-only rescoring methods, improving ASR accuracy.
Key insights
- Multi-modal LLMs, by jointly modeling speech and text, leverage cross-modal knowledge transfer to enhance ASR rescoring performance.
- The sequential order of speech and text tokens in the input sequence impacts performance, with a "speech-first" configuration generally being more effective.
- Even when only text tokens are used for rescoring, Speech-Text LLMs outperform text-only LLMs, indicating that pre-training on speech data improves their understanding of text for this task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.