Reproduce “DrawSpeech: Expressive Speech Synthesis Using Prosodic Sketches as Control Conditions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weidong Chen, Shan Yang, Guangzhi Li, Xixin Wu
Researchers from The Chinese University of Hong Kong and Tencent AI Lab developed DrawSpeech, a system that generates expressive speech by allowing users to visually sketch prosody trends. The system, built on a Latent Diffusion Model, achieved an MOS of 4.49 for naturalness and demonstrated precise, word-level control over pitch and energy, enabling intuitive and fine-grained adjustments.
Key insights
- Prosody sketches offer an intuitive, intermediate control mechanism, addressing limitations of both reference-based and natural language-based prosody control.
- Merely inputting prosody sketches into existing TTS architectures is insufficient for effective control and high naturalness; a specialized Sketch-to-Contour Predictor is crucial.
- User-guided prosody sketches can enhance the naturalness and expressiveness of synthetic speech, sometimes perceived as matching or slightly exceeding ground-truth human speech.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.