Reproduce “Effective Text Adaptation for LLM-based ASR through Soft Prompt Fine-Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yingyi Ma, Zhe Liu, Ozlem Kalinli
Researchers at Meta AI introduce a two-step soft prompt fine-tuning strategy for LLM-based Automatic Speech Recognition (ASR) to effectively adapt to entity-heavy domains using only text-only data. This method mitigates prompt mismatch during fine-tuning, resulting in relative Word Error Rate (WER) reductions up to 8.9% and Entity Error Rate (EER) reductions up to 18.0% across music and chatbot domains.
Key insights
- Direct text-only fine-tuning of LLM-based ASR decoders without explicit prompt management performs poorly due to the significant prompt mismatch.
- Learning a domain-specific soft prompt can effectively bridge the modality gap, enabling the LLM decoder to integrate new textual knowledge more efficiently during adaptation.
- The soft prompt strategy is complementary to external Language Model (LM) fusion, with both methods contributing independent and combined gains for domain adaptation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.