Reproduce “TuneJury: An Open Metric for Improving Music Generation Preference Alignment” on OpenResearch

About this paper

Yonghyun Kim, Junwon Lee, Haiwen Xia, Yinghao Ma, Junghyun Koo, Koichi Saito, Yuki Mitsufuji, Chris Donahue

TuneJury introduces an open, instance-level pairwise reward model designed to predict human music preference for text prompts and audio clips, improving alignment in music generation systems. The model achieves competitive performance on benchmarks, with a 0.7086 pairwise accuracy on its held-out test split, utilizing only human-rated data and a leaner architecture, and can be efficiently adapted to new systems using anchor calibration.

Key insights

• A leaner reward model architecture, with text and audio inputs only, can achieve competitive human preference prediction using solely human-provided supervision, without complex inputs or pseudo-label augmentation.
• Frozen reward models can effectively drive preference alignment in TTM systems across various development stages, including inference-time selection, latent optimization, and expert-iteration post-training.
• The observed 'reward exploitation' behavior in optimization tasks provides empirical evidence that TuneJury captures a genuine preference-alignment signal, even if it entails trade-offs with distributional fidelity.

Summary by alphaXiv.