Reproduce “Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Siddhant Arora, Haidar Khan, Kai Sun, Xin Luna Dong, Sajal Choudhary, Seungwhan Moon, Xinyuan Zhang, Adithya Sagar and 9 more
Researchers at Meta and Carnegie Mellon University developed "Stream RAG," a framework that integrates external tool usage directly into end-to-end speech-in speech-out systems to enhance factual accuracy and significantly reduce user-perceived latency. The approach, particularly the model-triggered variant, achieved over 200% relative accuracy improvement and up to 53.4% reduction in tool use latency on new AudioCRAG benchmarks.
Key insights
- Integrating external tools is essential for overcoming factual limitations and hallucinations in E2E spoken dialogue systems, but managing retrieval latency is critical for real-time interaction.
- A model-triggered approach for streaming RAG, which learns when to query based on partial speech, is more efficient and effective for minimizing user-perceived latency than fixed-interval querying.
- Specific post-training techniques, including error recovery via negative sampling, significantly improve the robustness and accuracy of streaming tool query generation and subsequent speech output.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.