Reproduce “Evaluating and Rewarding LALMs for Expressive Role-Play TTS via Mean Continuation Log-Probability” on OpenResearch

About this paper

Yong Ren, Jingbei Li, Haiyang Sun, Yujie Chen, Cheng Yi, Yechang Huang, Hao Gu, Ye Bai and 1 more

Researchers developed Mean Continuation Log-Probability (MCLP), an objective metric that quantifies stylistic consistency in speech by leveraging the in-context learning capabilities of Large Audio Language Models (LALMs). Integrating MCLP as a reward signal within a reinforcement learning framework improved Role-Play Text-to-Speech (RP-TTS) performance, achieving a subjective MOS of 3.576 and reducing Character Error Rate to 1.130%.

Key insights

• MCLP reliably correlates with human judgments of stylistic consistency, providing an objective and interpretable measure that leverages the implicit understanding of latent style spaces within pretrained LALMs.
• A hybrid reward combining stylistic consistency (MCLP) and content fidelity (CER) is crucial for effective RL in expressive TTS, successfully mitigating reward hacking while achieving balanced performance.
• Utilizing audio history in multi-turn dialogue significantly enhances stylistic continuity, and the proposed model effectively leverages these cues, unlike many baselines.

Summary by alphaXiv.