Reproduce “Drop the beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziqian Ning, Shuai Wang, Yuepeng Jiang, Jixun Yao, Lei He, Shifeng Pan, Jie Ding, Lei Xie
Freestyler is presented as the first system for generating rapping vocals from lyrics and accompanying music, designed to synchronize with the beat and control timbre via a reference audio. The system, supported by the newly created RapBank dataset, produces natural and rhythmically aligned rap vocals, achieving an MOS-N of 3.92 and MOS-M of 3.84, which are comparable to ground truth.
Key insights
- Conditioning on fine-grained accompaniment features is essential for generating natural, rhythmic rap vocals that are well-aligned with the accompanying beat, as evidenced by significant performance drops in ablation studies without it.
- Effective zero-shot timbre control requires multi-stage conditioning of the reference audio embedding in both the language model (lyrics-to-semantic) and the spectrogram generation (semantic-to-spectrogram) stages.
- A random masking strategy applied to accompaniment features during training is critical for mitigating train-inference length mismatches, preventing issues like early termination or incoherent output during inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.