Reproduce “DiaMoE-TTS: A Unified IPA-Based Dialect TTS Framework with Mixture-of-Experts and Parameter-Efficient Zero-Shot Adaptation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziqi Chen, Gongyu Chen, Yihua Wang, Chaofan Ding, Zihao chen, Wei-Qiang Zhang
Researchers from Tsinghua University and Giant Network AI Lab developed DIAMOE-TTS, a unified IPA-based text-to-speech framework for low-resource Chinese dialects, incorporating a Mixture-of-Experts module and parameter-efficient zero-shot adaptation. The framework demonstrated effective learning and natural speech generation for various dialects, including specialized styles like Peking Opera, through an open-data approach.
Key insights
- Unified IPA representation is crucial for overcoming inconsistent orthographies and ambiguous phonetic guidance in dialect TTS, leading to substantial improvements in speech quality and intelligibility.
- A dialect-aware Mixture-of-Experts (MoE) module effectively captures and preserves the unique phonological characteristics of multiple dialects, mitigating style averaging.
- Parameter-efficient adaptation techniques allow for rapid and effective transfer learning to extremely low-resource dialects with minimal data, expanding the framework's scalability and practical applicability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.