Reproduce “DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jianing Yang, Yusuke Fujita, Yui Sudo
DuplexCascade adds turn-taking control to a streaming speech pipeline by sending partial recognized text to a language model in short intervals. The model uses control tokens to wait, respond, or stop, while speech recognition and synthesis remain modular. On Full-Duplex-Bench, it reports averaged turn-taking accuracy of 0.858 versus 0.598 for MiniCPM-Duplex, alongside VoiceBench Overall of 65.41 versus 69.66.
Key insights
- Integrating conversational control tokens directly into the LLM allows for robust, LLM-driven turn-taking decisions without relying on external acoustic Voice Activity Detection.
- Adapting the LLM with text-only data via LoRA fine-tuning effectively preserves the core conversational intelligence of the backbone LLM, avoiding degradation from cross-modal learning.
- Micro-turn based interaction enables fine-grained, real-time control over dialogue flow and system response generation, supporting concurrent listening and speaking.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.