Reproduce “DurIAN: Duration Informed Attention Network For Multimodal Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng Liu, Deyi Tuo and 4 more
Tencent AI Lab's DurIAN framework generates robust and natural speech and synchronized facial expressions by employing a duration-informed alignment network, avoiding the common errors of end-to-end TTS systems. It incorporates an optimized Multi-band WaveRNN that accelerates inference speed by 7.8 times on a single CPU core without quality compromise.
Key insights
- Explicit duration modeling provides a robust alternative to end-to-end attention in neural TTS, effectively eliminating common synthesis errors while maintaining high naturalness.
- The combination of multi-band processing and 8-bit quantization can dramatically improve neural vocoder inference speed on standard CPUs without compromising perceived audio quality.
- A shared duration model enables flexible synchronization of unimodal generative models (e.g., speech and face), reducing the need for extensive parallel multimodal training datasets.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.