Reproduce “Intelligible Lip-to-Speech Synthesis with Speech Units” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jeongsoo Choi, Minsu Kim, Yong Man Ro
KAIST researchers developed a framework for intelligible Lip-to-Speech (L2S) synthesis that generates high-quality speech from lip movements without requiring text-labeled data, reducing the Word Error Rate to 50.2% on LRS3 and 44.8% on LRS2 datasets while achieving high subjective naturalness and intelligibility. This approach leverages self-supervised speech units as a pseudo-text target and employs a multi-input vocoder robust to imperfect L2S outputs.
Key insights
- Quantized self-supervised speech representations can effectively serve as a 'pseudo-text' target, offering powerful linguistic content supervision for L2S models without relying on costly human text labels.
- A vocoder explicitly designed to accept both continuous acoustic features (mel-spectrograms) and discrete linguistic units (speech units) can generate higher quality and more intelligible waveforms, especially when trained with augmented noisy inputs.
- Utilizing advanced pre-trained visual encoders, such as AV-HuBERT, provides significantly richer and more robust visual features, further boosting L2S performance to levels comparable with state-of-the-art lip-reading models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.