Reproduce “DelightfulTTS 2: End-to-End Speech Synthesis with Adversarial Vector-Quantized Auto-Encoders” on OpenResearch

About this paper

Yanqing Liu, Ruiqing Xue, Lei He, Xu Tan, Sheng Zhao

DelightfulTTS 2 introduces an end-to-end text-to-speech system that learns optimal speech representations directly from raw audio using a Vector-Quantized Generative Adversarial Network (VQ-GAN) and jointly optimizes all components. This system achieved a Mean Opinion Score of 4.26, closely approaching ground-truth naturalness, while maintaining efficient inference.

Key insights

• Learning speech representations directly from raw audio via VQ-GAN can capture richer acoustic details, including phase information, outperforming hand-crafted features like mel-spectrograms.
• Jointly optimizing the entire text-to-waveform pipeline (acoustic model predicting learned codes, codec decoding codes) eliminates training-inference mismatch and improves overall speech quality.
• Adversarial training with specialized discriminators (like DWT-based) is effective for learning high-fidelity, compact discrete speech representations.
• The success of learned speech representations suggests moving away from fixed, hand-crafted features toward adaptive, task-specific representations for speech processing tasks beyond synthesis.
• The demonstrated benefits of joint training indicate that future TTS systems should consider end-to-end optimization rather than component-wise training, even when computational complexity increases.

Summary by alphaXiv.