Reproduce “VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing” on OpenResearch

About this paper

Jiacheng Xu, Heting Gao, Liufei Xie, Zhenchuan Yang, Lijiang Li, Yiting Chen, Bin Zhang, Meng Chen and 3 more

VITA-QINYU is an end-to-end spoken language model developed by TME Lyra Lab and Tencent YouTu Lab that generates expressive speech for role-playing and singing, in addition to natural conversation. It integrates dynamic timbre control and achieves high character matching scores in role-playing and competitive singing quality compared to specialized systems, while also outperforming other SLMs on standard conversational benchmarks.

Key insights

• The hybrid speech-text paradigm effectively enhances paralinguistic expressivity by leveraging multi-codebook audio tokens while maintaining clear modality separation, reducing cross-modal interference.
• Explicit injection of speaker information and dynamic timbre control through the Text-to-Timbre module is crucial for achieving high character matching and diverse vocal styles in role-playing scenarios.
• Utilizing a multi-codebook XY-Tokenizer for speech encoding and decoding provides superior speech and singing voice reconstruction compared to single-codebook tokenizers, preserving melodic variations and overall quality.
• Large-scale, synthetically generated datasets for specific expressive tasks like role-playing and singing are critical for successfully training end-to-end spoken language models to perform these complex functions.

Summary by alphaXiv.