Reproduce “Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations” on OpenResearch

About this paper

Guan-Ting Lin, Cheng-Han Chiang, Hung-yi Lee

Researchers from National Taiwan University present the StyleTalk dataset and Spoken-LLM framework to enable large language models to interpret varied speaking styles in spoken conversations and generate stylistically appropriate responses. The work demonstrates that incorporating paralinguistic and prosodic cues significantly enhances an LLM's ability to produce more reasonable and diverse spoken outputs.

Key insights

• Incorporating explicit speaking style information significantly enhances an LLM's ability to generate stylistically appropriate and diverse spoken responses.
• Human filtering is crucial for creating high-quality, natural multimodal dialogue datasets, as automatically generated data (even from LLMs) often lacks the nuanced human perception of style-dependent responses.
• A two-stage training approach, starting with warmup pre-training on larger, lower-quality data and then fine-tuning on smaller, high-quality human-filtered data, is effective for developing robust multimodal LLMs with limited human-annotated resources.

Summary by alphaXiv.