Reproduce “Revisiting IPA-based Cross-lingual Text-to-speech” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haitong Zhang, Haoyue Zhan, Yang Zhang, Xinyuan Yu, Yue Lin
An empirical study on IPA-based cross-lingual Text-to-Speech revealed that language-unique phonetic and prosodic features act as channels for speaker information leakage, hindering effective cross-lingual voice cloning. Increasing the number of speakers in a balanced manner across all languages within the training dataset effectively mitigates this issue, leading to improved voice cloning performance and naturalness.
Key insights
- The specific architectural choices for processing IPA and suprasegmental input embeddings (SEA vs. UEI) have a negligible impact on cross-lingual voice cloning performance.
- Language-unique IPA symbols and suprasegmental features (tones and stresses) act as 'leakage channels' for speaker-specific information, hindering effective cross-lingual voice cloning.
- Achieving robust cross-lingual voice cloning requires increasing the number of speakers in the training data, with a balanced distribution across all target languages being critical for optimal results.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.