Reproduce “Speech Retrieval-Augmented Generation without Automatic Speech Recognition” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Do June Min, Karel Mundnich, Andy Lapastora, Erfan Soltanmohammadi, Srikanth Ronanki, Kyu Han

Researchers at AWS AI Labs and the University of Michigan developed SpeechRAG, an end-to-end framework for open-question answering directly from spoken data, bypassing Automatic Speech Recognition. This system's retrieval component achieves performance on par with or exceeding systems using perfect text transcripts and consistently outperforms ASR-dependent baselines in high-error conditions, effectively mitigating error propagation.

Key insights

  • Robust speech retrieval can be achieved by distilling knowledge from a frozen text-based retriever into a lightweight speech adapter, avoiding extensive cross-modal pre-training.
  • Directly processing speech for retrieval and generation effectively mitigates ASR error propagation, offering a more robust approach for spoken content understanding.
  • Operating directly on audio can preserve more information than lossy text transcriptions, though current Speech Language Models (SLMs) may struggle with very long audio contexts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.