Reproduce “Content is What Remains: Invariant Speech Tokenization from Parallel Utterances” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Laurin Wagner, Bernhard Thallinger, Miroslav Stankovic, Mario Zusag
PINT introduces a method for discrete speech tokenization that disentangles linguistic content from non-linguistic acoustic variations by leveraging parallel utterances and aggressive augmentations. This approach yields tokens with near-complete speaker invariance, leading to a 27-30% reduction in language model perplexity and compression rates approaching character-level text while retaining frame-level alignment.
Key insights
- Linguistic content is the singular common factor across parallel utterances, providing a strong signal to distill truly invariant semantic representations from acoustic variations.
- Effective semantic tokens require a dual criterion: robust capture of phonetic information and near-complete invariance to non-linguistic factors like speaker identity and noise.
- Jointly fine-tuning the encoder with alignment losses across parallel utterances and aggressive augmentations can effectively disentangle content from nuisance factors at both continuous and discrete levels.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.