Reproduce “Switchboard-Affect: Emotion Perception Labels from Conversational Speech” on OpenResearch

About this paper

Amrit Romana, Jaya Narain, Tien Dung Tran, Andrea Davis, Jason Fong, Ramya Rasipuram, Vikramjit Mitra

Researchers at Apple developed and released Switchboard-Affect, a new dataset providing comprehensive emotion perception labels for naturalistic conversational speech, addressing a critical data gap in Speech Emotion Recognition (SER). The dataset facilitates the training of more robust SER models capable of detecting subtle, spontaneous emotions and offers insights into the lexical and paralinguistic cues involved in human emotion perception.

Key insights

• Emotion perception in naturalistic conversational speech often involves subtle, lower-intensity expressions, leading to moderate inter-annotator agreement (Krippendorff’s alpha 0.25 for primary emotions) but meaningful co-occurrence patterns suggesting ambiguity rather than arbitrary labeling.
• Lexical content can be highly indicative for some emotions (e.g., fear, detected by GPT-4o with 0.586 probability), while paralinguistic cues are more crucial for others like calmness, tenderness, and happiness in spontaneous speech (lexical detection probabilities below 0.18).
• State-of-the-art SER models exhibit significant domain shift when evaluating naturalistic conversational speech, with varying performance across emotions (e.g., higher accuracy for sadness and happiness but lower for anger on Switchboard-Affect compared to MSP-Podcast).

Summary by alphaXiv.