Reproduce “$S^3$-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Heyang Liu, Jiayi Huang, Wenyang Xiao, Ziyang Cheng, Lixin Zhang, Zhen Liu, Miao He, Ronghua Wu and 3 more

S³-Bench evaluates whether speech interaction models can serve as reliable scientific voice assistants, rather than merely general conversational systems. Across ten scientific domains, its staged assessment reveals that models often preserve broad semantic context but struggle with technical terminology, abbreviations, symbolic expressions, strict factuality, and accurate spoken realization; the strongest systems reach 63.62% knowledge accuracy, 97.42% concept relevance, and roughly 5.7–5.9 turns per completed dialogue.

Key insights

  • Semantic recovery is considerably stronger than exact recognition: most models exceed 85% on relevant-concept detection, while leading terminology error rates remain above 36% and strict abbreviation errors are substantially higher than loose errors.
  • Speech-specific formatting can block access to otherwise available knowledge; special expressions and open-ended questions are especially difficult, with many spoken systems scoring below 20% on open-ended answers.
  • Reasoning quality and pronunciation quality are only weakly coupled: stronger scientific responses do not necessarily produce clearer technical speech, making the speech decoder an independent bottleneck.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.