Reproduce “UnifySpeech: A Unified Framework for Zero-shot Text-to-Speech and Voice Conversion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haogeng Liu, Tao Wang, Ruibo Fu, Jiangyan Yi, Zhengqi Wen, Jianhua Tao
UnifySpeech presents the first unified framework integrating zero-shot Text-to-Speech and Voice Conversion into a single system, demonstrating improved performance for both tasks through shared representations and a novel content alignment mechanism. The framework achieves this by using vector quantization and domain constraints to bridge the content domain gap between text-derived and speech-derived linguistic information.
Key insights
- Zero-shot Text-to-Speech (TTS) and Voice Conversion (VC) can be effectively unified into a single framework despite their different input modalities.
- Joint training provides mutual benefits: TTS gains enhanced speaker modeling from VC's diverse unlabeled data, and VC achieves better content-speaker disentanglement through TTS's text-based regularization.
- Vector quantization with a shared codebook and a domain constraint loss successfully bridges the content domain gap between text-derived and speech-derived linguistic information.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.