Reproduce “LA-RAG:Enhancing LLM-based ASR Accuracy with Retrieval-Augmented Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shaojun Li, Hengchao Shang, Daimeng Wei, Jiaxin Guo, Zongyao Li, Xianghui He, Min Zhang, Hao Yang
Researchers from Huawei TSC developed LA-RAG, a retrieval-augmented generation (RAG) framework for LLM-based Automatic Speech Recognition (ASR). It uses fine-grained speech-to-speech token retrieval to enhance accuracy, achieving an average Character Error Rate (CER) of 25.03% across Mandarin and Chinese dialect datasets, with notable improvements for accented speech.
Key insights
- Fine-grained, token-level speech-to-speech retrieval is crucial for effectively leveraging external knowledge to correct ASR errors, outperforming coarser or text-based retrieval methods.
- LLMs can enhance ASR accuracy, especially for accented speech, by learning mapping relationships between diverse pronunciations and correct tokens from retrieved in-context examples.
- Strategically pruning query tokens that are likely correct helps the LLM focus its attention on potentially erroneous or ambiguous segments, improving overall performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.