Reproduce “Speech Emotion Recognition Leveraging OpenAI's Whisper Representations and Attentive Pooling Methods” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ali Shendabadi, Parnia Izadirad, Mostafa Salehi, Mahmoud Bijankhan

Researchers from the University of Tehran developed a Speech Emotion Recognition framework utilizing frozen OpenAI Whisper representations combined with novel multi-head attention-based pooling methods. This approach achieved a new state-of-the-art unweighted accuracy of 83.07% on the Persian ShEMO dataset using a Whisper Small model, demonstrating competitive performance on English IEMOCAP with significantly fewer parameters than prior large models.

Key insights

  • Attention-based pooling mechanisms are more effective than global mean pooling at preserving emotion-relevant information within high-dimensional speech representations.
  • Larger Whisper models (e.g., Small vs. Tiny) provide richer representations, leading to improved classification, particularly for less frequent emotion categories.
  • The optimal Whisper encoder layer for extracting emotion-relevant features can vary by language and dataset, with intermediate layers sometimes outperforming the final layer, especially for lower-resource languages like Persian.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.