Reproduce “Advanced Zero-Shot Text-to-Speech for Background Removal and Preservation with Controllable Masked Speech Prediction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Leying Zhang, Wangyou Zhang, Zhengyang Chen, Yanmin Qian
Researchers at Shanghai Jiao Tong University developed a zero-shot Text-to-Speech (TTS) framework capable of both removing and preserving background acoustics from speech prompts in a controllable manner. The system utilizes a Controllable Masked Speech Prediction (CMSP) strategy and a Dual Speaker Encoder, achieving improved speech quality and consistent background management across diverse acoustic conditions.
Key insights
- Introducing a binary control signal enables a single TTS model to effectively and controllably perform the distinct tasks of background removal or preservation from a given prompt.
- A dual speaker encoder architecture significantly enhances the robustness of speaker timbre extraction and reduces interference between conflicting objectives in multi-task learning for audio generation.
- Masked speech prediction, when trained with diverse augmented data, can intelligently differentiate and handle acoustic backgrounds, leading to more natural and contextually appropriate synthesized speech outputs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.