Reproduce “MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

He Zhang, Wenqian Cui, Haoning Xu, Xiaohui Li, Lei Zhu, Haoli Bai, Shaohua Ma, Irwin King

MTR-DuplexBench is introduced as a new benchmark for comprehensively evaluating Full-Duplex Speech Language Models (FD-SLMs) in multi-round conversations, integrating a turn segmentation methodology and assessing conversational features, dialogue quality, instruction following, and safety. Evaluations revealed performance degradation and increased latency in FD-SLMs over multiple rounds, with end-to-end models showing lower instruction-following success and sensitivity to accent variations.

Key insights

  • Current FD-SLMs exhibit consistent performance degradation (decreased success rates, increased latency) across various conversational features as the number of interaction rounds increases.
  • Higher full-duplex capabilities in models do not necessarily correlate with superior dialogue quality, with HD-SLMs often achieving better coherence and meaningfulness.
  • While safety capabilities remain robust across multi-round and interrupted interactions, instruction following, particularly in end-to-end FD-SLMs, significantly declines over successive turns.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.