Reproduce “TSELM: Target Speaker Extraction using Discrete Tokens and Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Beilong Tang, Bang Zeng, Ming Li
The TSELM framework introduces a target speaker extraction network that leverages discrete audio tokens, derived from multiple layers of the WavLM self-supervised learning model, and processes them using a language model. This approach demonstrates improved speech quality compared to a discriminative baseline, but shows lower speech intelligibility, highlighting the challenges of information loss in audio tokenization.
Key insights
- Concatenating reference speech around the mixture during encoding significantly improves target speaker extraction by guiding WavLM to prioritize the target speaker and leverage its denoising capabilities.
- Integrating information from multiple, diverse hidden layers of WavLM provides a more robust and error-tolerant representation for TSE compared to relying on single-layer outputs.
- Discretization of speech signals introduces inherent information loss, leading to a performance gap in speech intelligibility and speaker similarity when compared to continuous embedding-based methods.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.