Reproduce “VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenpeng Du, Yiwei Guo, Hankun Wang, Yifan Yang, Zhikang Niu, Shuai Wang, Hui Zhang, Xie Chen and 1 more
VALL-T introduces a decoder-only generative transducer for text-to-speech that integrates Transducer principles into a Transformer architecture, significantly reducing hallucination issues in synthesized speech. The model achieves a 28.3% relative reduction in Word Error Rate (WER) compared to VALL-E while maintaining high naturalness and improving speaker similarity in zero-shot adaptation.
Key insights
- Monotonic alignment can be effectively enforced within a unified decoder-only Transformer architecture by leveraging relative position embeddings for input phonemes and a blank token mechanism, addressing hallucination without explicit duration prediction.
- Integrating Transducer principles into a decoder-only Transformer enables a 'best-of-both-worlds' solution, combining high naturalness and zero-shot speaker adaptation with robust, monotonically aligned speech generation.
- The proposed architecture not only reduces speech synthesis errors but also offers greater decoding control and inherent capabilities for forced alignment between phoneme and speech sequences.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.