Reproduce “Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Huaxuan Wang, Huimin Wang, Ruiyu Zhang, Yingjie Li, Yitao Duan

Confucius4-TTS introduces a zero-shot TTS system that performs transcript-free cross-lingual voice cloning across 14 languages. It achieves competitive intelligibility and speaker similarity, demonstrating an average WER of 3.73% on the CV3-Eval benchmark and strong human evaluation results across various cross-lingual settings.

Key insights

  • Jointly training a speaker encoder on self-supervised speech representations allows for robust, transcript-free speaker conditioning by extracting timbre features independent of explicit linguistic content.
  • A dual-mode conditioning strategy provides operational flexibility, enabling broad applicability through transcript-free cloning while offering improved speaker similarity when transcripts are available.
  • Leveraging an LLM-based Text-to-Semantic module and a conditional flow-matching Semantic-to-Acoustic module facilitates high intelligibility and speaker similarity across a wide range of languages.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.