Reproduce “Enabling Real-Time Conversations with Minimal Training Costs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wang Xu, Shuo Wang, Weilin Zhao, Xu Han, Yukun Yan, Yudi Zhang, Zhe Tao, Zhiyuan Liu and 1 more
A novel DUplex decOding (DUO) method allows Large Language Models to engage in real-time, interruptible conversations by simultaneously processing output and new user input. This approach drastically reduces training costs, requiring 500 times less data and significantly less computational power than prior duplex models, while achieving superior human-likeness and preserving general LLM performance.
Key insights
- Channel-division-multiplexing, using FlexAttention and KV-cache management, enables concurrent processing of input and output without increasing forward passes or inter-channel attention.
- The model learns to identify whether new user input is a query or non-query speech via lightweight state tokens, allowing for dynamic and intelligent conversational flow management.
- Achieving duplex capabilities primarily through decoding strategy optimization and minimal state token training is vastly more resource-efficient than prior architectural overhauls or time-division multiplexing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.