Reproduce “TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Giorgia Adorni, Michela Papandrea, Battista Rimoldi, Tiziano Leidi
TTM-Bench introduces a benchmarking framework for comparing text-to-music systems with different input formats, architectures, and local or API-based access. It evaluates musical-content alignment and computational efficiency separately, and a study of 13 systems, 100 reference tracks, and 26,000 generated clips shows that the strongest alignment systems are not necessarily the fastest or least expensive.
Key insights
- Semantic, genre, and descriptor alignment capture different properties: their generation-level correlations are weak, supporting the practice of reporting each component alongside the aggregate MCAS.
- System rankings are generally stable under alternative MCAS weightings, but close comparisons are uncertain; bootstrap intervals did not reliably distinguish several neighboring systems.
- Musical alignment and computational efficiency show no systematic relationship, so quality and resource demand should be treated as separate evaluation dimensions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.