Reproduce “MoshiRAG: Asynchronous Knowledge Retrieval for Full-Duplex Speech Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chung-Ming Chien, Manu Orsini, Eugene Kharitonov, Neil Zeghidour, Karen Livescu, Alexandre Défossez
MoshiRAG, developed by researchers from Toyota Technological Institute at Chicago and Kyutai, introduces the first full-duplex speech language model with Retrieval-Augmented Generation (RAG) capabilities. The system asynchronously retrieves external knowledge during natural conversational pauses, enhancing factual accuracy on knowledge-intensive tasks while preserving real-time interactivity.
Key insights
- Asynchronous RAG can be successfully integrated into full-duplex speech models by exploiting inherent conversational pauses, such as the delay before key information in a response.
- Training with synthetically generated multi-turn conversational data, structured with 'lead' and 'body' segments, is effective for teaching the model when and how to integrate retrieved information.
- The RAG mechanism enables the full-duplex model to generalize to out-of-domain tasks, like mathematical reasoning, demonstrating its potential for broader tool-use capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.