Reproduce “Deep Voice 2: Multi-Speaker Neural Text-to-Speech” on OpenResearch

About this paper

Sercan Arik, Gregory Diamos, Andrew Gibiansky, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, Yanqi Zhou

Researchers at Baidu AI Lab developed an all-neural multi-speaker text-to-speech system, Deep Voice 2, and augmented Tacotron, leveraging trainable speaker embeddings to synthesize high-quality speech from multiple distinct voices. The system achieved a mean opinion score (MOS) of 3.53 on the VCTK dataset with an 80-layer WaveNet, demonstrating accurate preservation of speaker identity with up to 100% classification accuracy and requiring less than half an hour of data per speaker.

Key insights

• Trainable, low-dimensional speaker embeddings, learned jointly with the TTS model parameters, are highly effective in enabling multi-speaker speech synthesis and preserving individual speaker identities.
• Replacing the Griffin-Lim algorithm with a WaveNet-based neural vocoder substantially improves the perceptual audio quality of spectrogram-based TTS models like Tacotron.
• The learned speaker embeddings reveal a meaningful latent space structure, capable of differentiating speakers by attributes such as gender and regional accent.

Summary by alphaXiv.