Reproduce “FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tian-Hao Zhang, Jiawei Zhang, Jun Wang, Xinyuan Qian, Xu-Cheng Yin
Researchers from USTB and Tencent AI Lab developed FaceSpeak, a method that synthesizes expressive and high-quality speech from human portraits of diverse artistic styles by disentangling identity and emotion features from visual input. The work also introduces the EM2TTS dataset to support multi-style multimodal speech synthesis.
Key insights
- Explicitly disentangling identity and emotion features from diverse visual prompts is crucial for achieving granular and flexible control over synthesized speech characteristics, moving beyond entangled representations.
- Developing a specialized, multi-style multimodal dataset, like EM2TTS, is essential to train models that can generalize vision-driven speech synthesis to non-photorealistic or stylized character portraits.
- Integrating advanced multi-modal feature extraction with dedicated disentanglement mechanisms and a state-of-the-art TTS backbone allows for robust and expressive speech generation aligned with arbitrary portrait styles.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.