Reproduce “OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yong Ren, Jiangyan Yi, Jianhua Tao, Haiyang Sun, Zhengqi Wen, Hao Gu, Le Xu, Ye Bai
Researchers from the Chinese Academy of Sciences and Tsinghua University developed OV-InstructTTS-TEP, a reasoning-driven framework that enables text-to-speech models to synthesize speech from open-vocabulary natural language instructions. The system, leveraging a new OV-Speech dataset, achieves superior instruction-following with a Gemini Score of 70.42, and produces highly natural and consistent speech with an MOS of 4.28 and ICMOS of 3.91.
Key insights
- Explicit reasoning, through a 'thinking' step, is crucial for bridging the semantic gap between abstract open-vocabulary instructions and concrete acoustic realizations, improving instruction following and generalization.
- The synergistic combination of explicit reasoning with paralinguistic-aware and emotion-attributed transcriptions (
- TEP
- components) is vital for achieving highly expressive and consistent speech synthesis from complex natural language inputs.
- A comprehensive, multi-layered dataset like OV-Speech, containing open-vocabulary instructions and LLM-generated reasoning chains, is essential for training models capable of interpreting and acting upon nuanced speech control cues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.