Reproduce “Faces that Speak: Jointly Synthesising Talking Face and Speech from Text” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Youngjoon Jang, Ji-Hoon Kim, Junseok Ahn, Doyeop Kwak, Hong-Sun Yang, Yoon-Cheol Ju, Il-Hwan Kim, Byeong-Yeol Kim and 1 more
KAIST and 42dot Inc. developed Text-to-Speaking Face (TTSF), a unified framework that simultaneously generates natural talking face videos and high-quality speech directly from text input. The system outperforms state-of-the-art cascade methods in video quality (FID 18.348), identity preservation (ID-SIM 0.864), and user perception, while ensuring consistent voice identity by effectively disentangling facial motion.
Key insights
- Integrating TFG and TTS into a unified framework allows for leveraging mutually complementary elements, overcoming the limitations of cascade systems and significantly improving overall efficiency and coherence.
- Disentangling identity features from dynamic motion in facial images is crucial for producing consistent and high-quality face-stylised speech, ensuring voice characteristics remain stable despite visual variations.
- Combining a sophisticated motion sampling technique (OT-CFM) with a dedicated temporal smoothing mechanism (auto-encoder normalizer) can generate a diverse range of natural and temporally coherent head poses.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.