Reproduce “MusicSem: A Semantically Rich Language--Audio Dataset of Natural Music Descriptions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rebecca Salganik, Teng Tu, Fei-Yueh Chen, Xiaohao Liu, Keifeng Lu, Ethan Luvisia, Zhiyao Duan, Guillaume Salha-Galvan and 3 more
A new language–audio dataset, MusicSem, captures a wider range of musical semantics from organic Reddit discussions, featuring 32,493 pairs and highlighting current multimodal models' limited sensitivity to nuanced human music descriptions. Fine-tuning models with MusicSem demonstrably enhances their semantic awareness, improving contextual and situational understanding by over 48% and 28% respectively.
Key insights
- Current state-of-the-art multimodal music models exhibit limited semantic sensitivity, showing greater awareness of descriptive changes than shifts in atmospheric, situational, or contextual semantics.
- MusicSem serves as a challenging benchmark, with models generally performing worse on it compared to other datasets, indicating a struggle to capture its richer and more nuanced textual semantics.
- Preliminary fine-tuning experiments demonstrate that MusicSem acts as an effective supervision signal, significantly enhancing models' semantic awareness and leading to improved performance on downstream tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.