Reproduce “VoxRAG: A Step Toward Transcription-Free RAG Systems in Spoken Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zackary Rackauckas, Julia Hirschberg
VoxRAG, developed at Columbia University, introduces a transcription-free retrieval-augmented generation system for spoken question answering, bypassing traditional ASR by directly retrieving audio segments using CLAP embeddings. The system achieves Recall@10 of 0.34 for very relevant segments and demonstrates the feasibility of direct audio-to-audio retrieval for generating answers from podcast content.
Key insights
- CLAP embeddings enable effective semantic matching between spoken queries and audio segments, allowing for direct audio-to-audio retrieval without ASR.
- While transcription-free retrieval shows promise for relevance, current methods face challenges in achieving high accuracy and completeness for precise factual answers.
- Reranking mechanisms like cross-encoders, in their current application, did not substantially improve retrieval performance for audio content over basic cosine similarity.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.