Reproduce “MusicEval: A Generative Music Dataset with Expert Ratings for Automatic Text-to-Music Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cheng Liu, Hui Wang, Jinghua Zhao, Shiwan Zhao, Hui Bu, Xin Xu, Jiaming Zhou, Haoqin Sun and 1 more
The research presents MusicEval, the first generative music dataset featuring expert ratings for text-to-music (TTM) evaluation, and establishes an automatic assessment task. It demonstrates the feasibility of using a CLAP-based model to predict musical impression and textual alignment scores with strong correlation to human expert judgments.
Key insights
- Creating a diverse dataset with expert ratings, covering various generative models and prompts, is critical for training effective automatic TTM evaluation systems.
- Fine-tuned pre-trained models like CLAP can learn representations that strongly correlate with human expert judgments of both musical quality and text-music alignment.
- While predicting 'musical impression' showed stronger correlations, 'textual alignment' also demonstrated significant system-level alignment with human scores, affirming the task's feasibility.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.