Reproduce “Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiudong Yang, Peiying Wang, Yi Zhu, Mingchao Feng, Meng Chen, Xiaodong He
Researchers at JD AI developed a Gated Multimodal Fusion (GMF) model with contrastive learning for turn-taking prediction in human-robot dialogue. The approach, leveraging a new large-scale, real-world multimodal dataset, achieved improved Macro-F1 scores for both user endpointing and barge-in detection tasks.
Key insights
- Semantic, acoustic, and timing features are complementary, and their combined integration is crucial for accurate turn-taking prediction, with semantic features being particularly influential.
- The proposed gated multimodal fusion mechanism effectively controls and combines contributions from different modalities, outperforming simpler fusion strategies and other advanced methods.
- Data augmentation via contrastive learning effectively addresses class imbalance in turn-taking datasets, leading to improved performance, especially for minority classes, and more robust feature representations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.