Reproduce “DiscreTalk: Text-to-Speech as a Machine Translation Problem” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tomoki Hayashi, Shinji Watanabe
DiscreTalk reconceptualizes Text-to-Speech (TTS) as a Neural Machine Translation problem, directly translating text into discrete speech symbols learned by a VQ-VAE, then reconstructing audio. The system achieves a naturalness Mean Opinion Score (MOS) of 3.93, surpassing a Transformer-TTS baseline's 3.48 MOS and eliminating reliance on handcrafted acoustic features.
Key insights
- Smaller VQ-VAE downsampling factors (e.g., DSF128) yield better speech articulation and synthesis performance compared to larger factors (e.g., DSF256).
- Subword units are crucial for compressing high-resolution discrete speech representations, preventing generation failures and improving synthesis quality.
- The system exhibits a low correlation between NMT token error rates and speech intelligibility, suggesting a one-to-many mapping where multiple discrete sequences can represent the same text.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.