Reproduce “ProsoCodec: Prosody-Oriented Speech Codec for Voice Conversion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jeongsoo Choi, Ji-Hoon Kim, Shujie Hu, Joon Son Chung
ProsoCodec introduces a prosody-oriented neural speech codec for voice conversion, leveraging a diffusion autoencoder that models prosody as a conditional residual based on explicit text and speaker priors. The model achieves superior performance in zero-shot voice conversion, exhibiting the lowest WER (4.451), highest P-MOS (3.852), highest target speaker similarity (SIM_r 0.565), and lowest source-timbre leakage (SIM_s 0.167) among compared state-of-the-art methods.
Key insights
- Modeling prosody as a conditional residual, by explicitly accounting for content and speaker identity, offers an effective and stable alternative to complex disentanglement methods for capturing fine-grained prosodic information.
- The dual-utterance training strategy is crucial for mitigating prompt-style leakage, enabling robust preservation of source prosody during voice conversion.
- Biasing the encoder input to low-frequency mel bands effectively directs the codec's discrete bottleneck to encode prosodic variations, which primarily reside in these acoustic regions, improving control and disentanglement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.