Reproduce “Multimodal Music Recommendation System using LLMs” on OpenResearch

About this paper

Srikar Prabhas Kandagatla, Sreehitha R. Narayana, Chandana Magapu, Swetha Mohan, Shamanth Kuthpadi, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt and 1 more

Researchers from the University of Massachusetts, Amherst, Dolby Laboratories, Adobe Research, and Cisco Research developed a multimodal music recommendation framework that enriches song representations with audio, lyric, and LLM-generated semantic metadata, alongside listening completion ratios. Integrating these into an extended LLM-based sequential recommender, the system achieved up to 95% Recall and 79% NDCG gains over ID-only baselines in fine-tuned settings, providing a new public benchmark for content-aware music recommendation.

Key insights

• Fine-tuning LLMs (e.g., with LoRA) is essential for multimodal content features to consistently enhance recommendation quality, significantly outperforming zero-shot configurations.
• LLM-generated semantic metadata, particularly when structured like MGPHot annotations, offers more valuable and interpretable signals than other metadata sources.
• Optimal performance depends on the selection of specific modality embedders, with EnCodec showing strong results for audio and MPNet for lyrics.

Summary by alphaXiv.