Reproduce “DanceMeld: Unraveling Dance Phrases with Hierarchical Latent Codes for Music-to-Dance Synthesis” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xin Gao, Li Hu, Peng Zhang, Bang Zhang, Liefeng Bo
DanceMeld introduces a two-stage framework for music-to-dance synthesis by decoupling dance motions into hierarchical latent codes for poses and movements. The system leverages a hierarchical VQ-VAE and a diffusion model to generate physically plausible and music-aligned dance sequences, enabling applications like dance style transfer and unit editing.
Key insights
- Decoupling dance into hierarchical latent codes (poses and movements) provides an interpretable and controllable representation that aligns with professional choreography principles.
- Using a single Gaussian diffusion model to predict the joint distribution of continuous hierarchical latent features, conditioned on music, yields superior generation quality compared to discrete VQ-Diffusion or separate diffusion models.
- Incorporating specific auxiliary losses for motion dynamics and foot contact, alongside a modality alignment loss with music features, is crucial for achieving high physical realism and beat synchronization in generated dance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.