Reproduce “Data Efficient Voice Cloning for Neural Singing Synthesis” on OpenResearch

About this paper

Merlijn Blaauw, Jordi Bonada, Ryunosuke Daido

This research demonstrates that neural singing synthesis can generate high-quality singing voices with as little as 2 minutes of target singer data by adapting voice cloning techniques from text-to-speech systems. The work shows that multispeaker models trained on diverse singing data enable efficient voice cloning, achieving synthesis quality comparable to models trained on significantly larger datasets.

Key insights

• Modern voice cloning techniques, successful in text-to-speech, can be effectively adapted for neural singing synthesis, significantly reducing data requirements.
• Training multispeaker models on 'pseudo singing' data (phrases with consistent pitch/cadence) creates a more homogeneous base voice, leading to better adaptation results for new singers.
• While pseudo singing is effective for base model training, adaptation using natural, expressive singing data yields synthetic voices that listeners strongly prefer for their expressiveness.

Summary by alphaXiv.