Reproduce “Cross-Lingual F5-TTS: Towards Language-Agnostic Voice Cloning and Speech Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qingyu Liu, Yushen Chen, Zhikang Niu, Chunhui Wang, Yunting Yang, Bowen Zhang, Jian Zhao, Pengcheng Zhu and 2 more
Researchers from Shanghai Jiao Tong University, Johns Hopkins University, and Geely developed Cross-Lingual F5-TTS, a flow-matching-based model for cross-lingual zero-shot voice cloning and speech synthesis that operates without requiring transcripts for reference audio prompts. This framework achieved comparable intra-lingual quality to the original F5-TTS and effectively cloned voices across unseen languages by integrating a novel speaking rate predictor.
Key insights
- Flow-matching-based TTS models can be successfully adapted for transcript-free, cross-lingual voice cloning while preserving competitive intra-lingual performance.
- Accurate duration modeling, decoupled from reference text transcripts, is essential for high-quality cross-lingual speech synthesis, with optimal linguistic granularity for prediction varying by language (e.g., phonemes for English, syllables for Chinese).
- The proposed speaking rate predictor exhibits a degree of generalization, enabling cross-lingual voice cloning for unseen prompt languages (German, French, Hindi, Korean) despite being trained exclusively on Chinese and English data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.