Reproduce “Prompting Large Language Models with Speech Recognition Abilities” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong and 4 more
Researchers at Meta AI developed a method to enable Large Language Models to perform multilingual Automatic Speech Recognition by prepending audio embeddings to text inputs, achieving an average Word Error Rate of 9.7% on MLS and outperforming prior baselines. The approach demonstrates that robust ASR can be achieved even with a frozen LLM, preserving its core capabilities.
Key insights
- Large Language Models can effectively perform Automatic Speech Recognition (ASR) by simply prepending audio embeddings to their text inputs, effectively prompting them with speech.
- Robust multilingual ASR can be achieved even with a completely frozen LLM (R=0 LoRA), suggesting that LLMs' original text-based capabilities can be preserved while adding new modalities.
- High audio encoder strides, compressing audio sequences by up to 96 times, still yield competitive ASR performance, indicating potential for processing extremely long audio inputs efficiently.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.