Reproduce “StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuhan Song, Linhao Zhang, Chuhan Wu, Aiwei Liu, Wei Jia, Houfeng Wang, Xiao Zhou
StableToken is introduced as a noise-robust semantic speech tokenizer for SpeechLLMs, featuring a multi-branch architecture with a consensus-driven mechanism to achieve stable token sequences under acoustic perturbations. The tokenizer exhibits an average Unit Edit Distance of 10.17% on noise-perturbed audio and enhances downstream SpeechLLM performance across ASR, SER, and TTS tasks in noisy environments.
Key insights
- A multi-branch, bit-wise voting architecture provides intrinsic robustness to quantization, enabling error correction at the bit level even if individual branches are perturbed by noise.
- Explicitly training for representational invariance by selectively exposing branches to clean and noisy inputs, combined with a consensus loss, effectively teaches the tokenizer to filter out acoustic perturbations.
- Improving the fundamental stability of speech tokens at the tokenizer level directly translates into substantial performance gains for various downstream SpeechLLM tasks, without incurring significant inference overhead.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.