Reproduce “Rethinking Music Captioning with Music Metadata LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Irmak Bukey, Zhepei Wang, Chris Donahue, Nicholas J. Bryan
Researchers at Carnegie Mellon University and Adobe Research introduce a two-stage "music metadata LLM" framework for music captioning, which first predicts structured music metadata from audio and then converts it into natural language captions using a pre-trained LLM. This approach achieves comparable performance to end-to-end models with less training, enables flexible post-training caption stylization, and supports metadata imputation.
Key insights
- Decoupling the factual content (metadata) from its linguistic expression (caption style) allows for greater flexibility and control over music caption generation.
- A two-stage audio-to-metadata-to-caption pipeline can achieve competitive performance with significantly reduced computational training costs compared to end-to-end models.
- Post-training prompt engineering on the metadata-to-caption conversion stage offers effective stylistic control over generated captions, a capability largely absent in end-to-end approaches.
- The model's ability to leverage partial metadata for imputation is valuable for enriching and organizing incomplete music databases, a function not supported by traditional end-to-end captioners.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.