Reproduce “Device Directedness with Contextual Cues for Spoken Dialog Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dhanush Bekal, Sundararajan Srinivasan, Sravan Bodapati, Srikanth Ronanki, Katrin Kirchhoff
Researchers at AWS AI Labs introduced a multi-modal model for barge-in verification in spoken dialogue systems, combining lexically-infused HuBERT speech representations with contextual dialogue cues. This system achieved an 81.3 F1 score, representing a 10.4% relative improvement over ASR-dependent baselines, with a 22% reduction in latency.
Key insights
- Fine-tuned self-supervised speech representations (HuBERT) implicitly encode sufficient linguistic information to outperform ASR-dependent methods for barge-in verification while reducing latency.
- Incorporating contextual cues, such as the chatbot's immediate prompt and dialogue context, significantly improves the accuracy of barge-in classification.
- Directly infusing lexical information into speech representations (LI-HuBERT) is an effective strategy for enhancing semantic understanding without incurring the computational overhead of runtime ASR.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.