Reproduce “SpeechSense: A Paralinguistic-Focused Dataset for Fine-Grained Speech Sentiment Analysis” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Shicheng Ma, Wenqian Cui, Irwin King

The SpeechSense dataset offers a resource for fine-grained speech sentiment analysis, specifically designed to train models on paralinguistic cues for nuanced interpersonal attitudes. Its construction enabled experiments demonstrating that acoustic input is essential for this task, with multi-modal language models achieving up to 57% F1 scores, a notable improvement over text-only approaches.

Key insights

  • Acoustic access is necessary for fine-grained attitudinal detection; text-only models consistently failed due to the semantic neutrality of the carrier text, often leading to severe mode collapse.
  • While dedicated speech encoders effectively decode attitudinal signals from prosody, multi-modal LLMs leveraging both acoustic and linguistic components achieve superior performance, indicating an enhanced reasoning capacity for nuanced distinctions.
  • The dataset reveals systematic confusion patterns between prosodically adjacent attitudes (e.g., Confident and Neutral), affirming that the defined categories isolate genuine prosodic cues rather than being ill-defined.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.