Reproduce “Everyone-Can-Sing: Zero-Shot Singing Voice Synthesis and Conversion with Speech Reference” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shuqi Dai, Yunyun Wang, Roger B. Dannenberg, Zeyu Jin

Carnegie Mellon and Adobe researchers introduce a groundbreaking unified framework for zero-shot singing voice synthesis and conversion that requires just 5 seconds of reference speech, achieving high-quality musical outputs through novel diffusion-based synthesis while addressing the critical challenge of limited singing training data through mixed speech-singing training strategies.

Key insights

  • Zero-shot synthesis can be achieved by effectively separating and recombining voice timbre, musical attributes, and pronunciation content
  • Speech data can effectively supplement limited singing data through careful training design and architecture choices

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.