Reproduce “Fluent but Unfeeling: The Emotional Blind Spots of Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Bangzhao Shu, Isha Joshi, Melissa Karnaze, Anh C. Pham, Ishita Kakkar, Sindhu Kothe, Arpine Hovasapian, Mai ElSherief

Researchers from Northeastern University and collaborators introduced EXPRESS, a benchmark utilizing self-disclosed human emotions from long-form social media texts to evaluate Large Language Models' (LLMs) fine-grained emotion recognition. The study found that while LLMs often struggle with precise emotional alignment, top models achieved average vector F1 scores up to 0.711, and few-shot prompting improved performance, demonstrating in-context learning for emotional tasks.

Key insights

  • Masked Language Models (e.g., RoBERTa, Longformer) performed comparably to or better than significantly larger causal LLMs (like GPT-4o) in zero-shot emotion recognition, suggesting architectural strengths for this task.
  • Chain-of-Thought (CoT) prompting consistently degraded emotion recognition performance across causal LLMs, indicating its unsuitability for subjective, context-sensitive emotional tasks.
  • LLMs demonstrated in-context learning capabilities, with few-shot prompting (especially with semantically nearest examples) significantly improving emotion recognition performance over zero-shot settings.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.