Reproduce “MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guobin Ma, Yuepeng Jiang, Dake Guo, Huakang Chen, Yuan Zhou, Lingxin Xiong, Wenkai Cheng, Jingbin Hu and 10 more
MSVAE organizes song audio into separate latent subspaces for melody and harmony, and lyric-related linguistic-vocal content. It aligns these streams during training to frozen musical and speech references, then supports lyrics-to-song generation from an untimed phoneme prefix. In the shared-generator setup, its phoneme error rate is 0.136 versus 0.851 for the best baseline, but timestamps lower it to 0.069.
Key insights
- Separating compositional and linguistic-vocal supervision into different latent subspaces can make musical content more accessible without treating singing and melody as separate audio sources.
- In the reported lyrics-to-song setup, organizing the representation supports lyric generation from untimed phoneme input, reducing dependence on sentence-level timestamps.
- The two latent halves show different behavior when selectively removed: one favors accompaniment and vocal melody, while the other preserves intelligible singing with sparser backing; this suggests specialization, not exclusive encoding.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.