Reproduce “VECL-TTS: Voice identity and Emotional style controllable Cross-Lingual Text-to-Speech” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ashishkumar Gudmalwar, Nirmesh Shah, Sai Akarsh, Pankaj Wasnik, Rajiv Ratn Shah
The VECL-TTS system, developed by Sony Research India and IIIT Delhi, enables simultaneous control of voice identity and emotional style in cross-lingual text-to-speech, achieving an average 8.83% relative improvement over state-of-the-art methods in preserving these attributes across English and three Indian languages. It improves emotion similarity (MOS 3.82) and maintains high naturalness and speaker similarity (MOS 3.81).
Key insights
- Explicitly conditioning phoneme duration prediction with both speaker and emotion features is crucial for generating stable and natural speech that accurately reflects the intended rhythm and emotional state.
- Disentangled multilingual speaker and emotion embedding networks, combined with tailored consistency losses, effectively enable robust cross-lingual transfer of both voice identity and emotional style.
- Leveraging self-supervised speech representations (wav2vec2) as a content loss during training effectively mitigates pronunciation degradation and improves speech quality during cross-lingual style transfer.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.