Reproduce “Zero-Shot Face-to-Speech Synthesis via Latent Space Adaptation of a Style-Diffusion TTS Model” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Carlos Muñoz-Romero, Jose A. Gonzalez-Lopez

Researchers from Universitat Oberta de Catalunya and University of Granada developed a "Freeze-Align" framework for zero-shot Face-to-Speech synthesis, adapting facial features to a frozen StyleTTS 2 model. The system synthesizes voices from static images, prevents mode collapse, and demonstrates cross-lingual transfer, achieving a Top-1 face-to-voice retrieval accuracy of 9.3% for unseen identities and high English naturalness (UTMOS 3.7-4.0).

Key insights

  • A discernible, albeit weak, biometric signal for speaker identity can be reliably extracted from static facial images and mapped to a sophisticated TTS model's style space without full model retraining.
  • Soft-tuning specific layers of the face encoder is crucial for effective visual-acoustic latent space alignment, allowing visual features to specialize towards vocally-relevant traits and improving synthesis quality.
  • The learned face-to-style mapping exhibits zero-shot cross-lingual generalization, enabling fluent speech synthesis in a new language (Spanish) using an adapter trained solely on another (English) without further multilingual data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.