Reproduce “Robust One-Shot Singing Voice Conversion” on OpenResearch

About this paper

Naoya Takahashi, Mayank Kumar Singh, Yuki Mitsufuji

Researchers at Sony Group Corporation developed a robust one-shot singing voice conversion (ROSVC) system capable of converting singing voices to unseen target singers while effectively handling distorted input audio. The system achieved superior pitch accuracy with a Pitch Mean Absolute Error (PMAE) of 8.6 for seen and 9.0 for unseen singers, alongside higher subjective naturalness and similarity scores compared to baseline models.

Key insights

• One-shot singing voice conversion can be effectively achieved by combining a domain-independent style encoder with explicit F0 conditioning, significantly improving pitch accuracy.
• Robustness to complex audio distortions can be systematically enhanced through a two-stage training approach where enhancement blocks are trained on distorted data while the core conversion model is frozen.
• Hierarchical multi-resolution diffusion models, conditioned on progressively lower sampling rate signals, provide a superior approach for synthesizing high-quality singing voices, addressing vibrato stability challenges.

Summary by alphaXiv.