Reproduce “PromptTTS: Controllable Text-to-Speech with Text Descriptions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, Xu Tan
PromptTTS, developed by a collaboration including Microsoft Research Asia, introduces an end-to-end system for controllable text-to-speech that synthesizes speech guided by natural language style descriptions. The system achieves an average style control accuracy of 89.57% on a new dataset, PromptSpeech, offering a user-friendly interface for expressive speech generation.
Key insights
- The end-to-end design of PromptTTS effectively mitigates the 'cascaded error' problem inherent in two-stage baseline systems, where errors from explicit style prediction propagate and degrade overall control accuracy.
- Fine-tuning the BERT-based style encoder on specific speech style factors is crucial for its ability to semantically interpret and utilize natural language style prompts, drastically improving performance over a generic pre-trained model.
- Natural language prompting provides a highly intuitive and accessible interface for controlling complex generative AI tasks, extending this paradigm from text and image generation to speech synthesis.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.