Reproduce “Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Minchan Kim, Myeonghun Jeong, Byoung Jin Choi, Semin Kim, Joun Yeop Lee, Nam Soo Kim
Researchers at Seoul National University and Samsung Research introduced a two-stage text-to-speech framework using neural transducers for semantic token prediction and a non-autoregressive speech generator. This framework achieved high-quality zero-shot adaptive speech synthesis with robust alignment and disentangled control over prosody and speaker identity.
Key insights
- Discrete semantic tokens derived from self-supervised speech representations can bridge the gap between continuous speech and neural transducers, enabling robust monotonic alignment in TTS.
- Decomposing the TTS pipeline into text-to-semantic token and semantic token-to-speech stages allows for specialized modeling, enhancing alignment robustness and leveraging non-autoregressive generation efficiency.
- Conditioning the token transducer and speech generator with separate reference speech streams enables disentangled control over prosody and speaker identity in the synthesized speech.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.