Reproduce “Sequential Trajectories and Simultaneous Blending: Multi-Emotion Modeling for Instruction-Following TTS” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yan Zhou, Yun Hong, Yang Feng
Researchers from the Chinese Academy of Sciences developed HybridEmo, a framework for Text-to-Speech (TTS) synthesis that generates speech with dynamic emotional expressions, encompassing both sequential emotion trajectories and simultaneous emotion blending. The system achieved improvements in macro emotion correctness for trajectories to 3.33 and emotion intensity for blending to 3.71 while maintaining speech intelligibility and speaker similarity.
Key insights
- Task-matched reinforcement learning reward functions are crucial for effectively guiding TTS models to generate complex multi-emotion patterns, overcoming the limitations of general SFT or RL.
- Approximating segment boundaries in speech based on text length and leveraging an emotion recognition model can provide effective, structure-aware feedback for sequential emotion trajectories without explicit timestamp annotations.
- A 'weaker-target margin' mechanism within the blending reward is effective in preventing single-emotion dominance, thereby fostering a more balanced and natural co-occurrence of multiple emotions in synthesized speech.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.