Reproduce “VoiceDesigner: Text-to-Voice Generation and Editing via Unified Diffusion Modeling and Data Augmentation” on OpenResearch

About this paper

Jiarui Hai, Karan Thakkar, Ke Chen, Yunyun Wang, Jiaqi Su, Rithesh Kumar, Mounya Elhilali, Zeyu Jin

VoiceDesigner introduces a unified diffusion model for text-to-voice generation and editing, leveraging a hybrid data augmentation pipeline and architectural innovations like token-level Adaptive Layer Normalization and 3D Rotary Positional Embeddings. The system achieves a 1.22% Word Error Rate and 0.66 Style-Accuracy in generation, and demonstrates superior performance in cloning diverse character voices and executing fine-grained edits compared to baselines.

Key insights

• Expanding training data with simulated diverse voices, including DSP-processed character voices and generatively augmented stylistic speech, is effective for overcoming data scarcity and broadening TTV capabilities.
• Explicitly differentiating noisy generation target tokens from clean conditioning tokens using token-level AdaLN and encoding modality-specific positional structures with 3D-RoPE significantly improves convergence and performance in multi-modal diffusion models.
• Unifying voice generation, cloning, and editing within a single framework streamlines development, reduces costs, and facilitates knowledge transfer between tasks, leading to more robust and versatile voice design tools.

Summary by alphaXiv.