Reproduce “DualTurn: Learning Turn-Taking from Dual-Channel Generative Speech Pretraining” on OpenResearch

About this paper

Shangeth Rajaa

DualTurn introduces a system that leverages dual-channel generative speech pretraining to enable anticipatory and nuanced turn-taking in modular conversational AI pipelines. It significantly enhances agent action prediction, achieving a weighted F1 of 0.633 on Switchboard, notably improving backchannel detection, and anticipates turn boundaries approximately 220 ms earlier than prior methods.

Key insights

• Generative speech pretraining on dual-channel conversational audio is critical for learning representations that enable accurate prediction of nuanced and sparse turn-taking events like backchannels.
• Semantic and prosodic information account for over 80% of the contribution to turn-end discrimination, significantly more than fine acoustic details.
• The pretrained transformer backbone develops multi-scale attention hierarchies, allowing it to attend to both short-range acoustic features and long-range conversational context, which is crucial for anticipation.

Summary by alphaXiv.