Reproduce “S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Feng Jiang, Zhiyu Lin, Yiyang Liu, Liumeng Xue, Fan Bu, Yuhao Du, Xiangying Chen, Benyou Wang and 1 more
S2S-Arena introduces a speech-native benchmark for evaluating speech-to-speech models, assessing both semantic instruction following and paralinguistic expression. The benchmark identifies a performance gap between industrial and academic models, with industrial systems significantly outperforming academic counterparts, especially in complex paralinguistic tasks, and highlights the importance of flow-matching based speech decoders for expressive generation.
Key insights
- A significant performance gap exists between industrial S2S systems (e.g., Qwen 2.5-Omni, GPT-4o-realtime, Doubao) and academic models, particularly as task complexity increases.
- Model performance is domain-sensitive, with some models excelling in knowledge-driven tasks (Education, Medical) and others in expressive domains (Entertainment, Social Interaction), suggesting unavoidable trade-offs.
- Architectural choices heavily influence performance; stronger backbone models improve instruction following, larger speech encoders enhance paralinguistic perception, and flow-matching-based speech decoders are critical for expressive speech generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.