Reproduce “DAIEN-TTS: Disentangled Audio Infilling for Environment-Aware Text-to-Speech Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ye-Xin Lu, Yu Gu, Kun Wei, Hui-Peng Du, Yang Ai, Zhen-Hua Ling
Researchers at the University of Science and Technology of China developed DAIEN-TTS, a zero-shot Text-to-Speech framework for independent control over speaker timbre and time-varying background environments. This system achieved high naturalness and strong speaker and environmental fidelity, comparable to human recordings, using a novel speech-environment separation module and cross-attention-based conditioning.
Key insights
- Effective disentanglement of speech and environment from mixed audio prompts using a dedicated separation module enables robust zero-shot TTS, even with noisy input conditions.
- A cross-attention-based conditioning mechanism is crucial for accurately modeling and reconstructing time-varying background environments, yielding superior naturalness and environmental fidelity compared to simpler concatenation methods.
- Combining Dual Classifier-Free Guidance with an SNR adaptation strategy provides fine-grained control over both speaker timbre and background environment characteristics, enhancing realism and controllability in synthesized speech.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.