Reproduce “Luna-TTS Family Technical Report” on OpenResearch

About this paper

Feng Yin, Shuai Shi, Junjie Zheng, Kechenying Zhou, Yiqiu Wang, Chenyang He, Qiuhua Jiang, Mengxiao Bi and 14 more

VUI Labs Research introduces the Luna-TTS Family, a suite of diffusion-language-model-based text-to-speech systems that offer high-quality and efficient speech generation for both offline and real-time applications. The systems, pretrained on 1 million hours of multilingual speech, demonstrate state-of-the-art zero-shot synthesis quality and exceptionally low first-block latency compared to existing models.

Key insights

• Masked acoustic-token diffusion is a viable and effective generative mechanism for production-scale TTS, offering inherent parallelism and infilling capabilities that address limitations of autoregressive models.
• A progressive adaptation recipe from a pretrained text LLM effectively transfers linguistic knowledge to speech diffusion models, enabling a unified architecture for both fully parallel and block-autoregressive decoding within a single model lineage.
• Reinforcement learning can be successfully applied to masked-diffusion models by defining policy ratios over the realized denoising trajectory, allowing for optimization based on utterance-level, non-differentiable speech feedback.

Summary by alphaXiv.