Reproduce “WildSpeech-Bench: Benchmarking End-to-End SpeechLLMs in the Wild” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Linhao Zhang, Jian Zhang, Bokai Lei, Chuhan Wu, Aiwei Liu, Wei Jia, Xiao Zhou
Tencent Inc. researchers developed WildSpeech-Bench, the first comprehensive benchmark for end-to-end Speech Large Language Models (SpeechLLMs) under real-world conditions, demonstrating that current models universally struggle with noise and speech-specific nuances, and S2T performance does not predict S2S capability.
Key insights
- Strong performance on Speech-to-Text (S2T) dialogue benchmarks does not translate to high performance in Speech-to-Speech (S2S) settings, indicating distinct challenges in multi-modal generation.
- All tested SpeechLLMs, including leading proprietary models like GPT-4o-Audio, exhibited a significant decline in performance when exposed to both natural and human background noise.
- The proposed query-aware evaluation framework demonstrated high alignment with human judgments, achieving a Pearson correlation of 0.86, significantly outperforming generic evaluation prompts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.