Reproduce “EmoBench-UA: A Benchmark Dataset for Emotion Detection in Ukrainian” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Daryna Dementieva, Nikolay Babakov, Alexander Fraser

Researchers at the Technical University of Munich and their collaborators created EMOBENCH-UA, the first manually annotated benchmark dataset for emotion detection in Ukrainian. This work provides a crucial resource for natural language processing in underrepresented languages and includes a comprehensive evaluation of diverse models, establishing baselines for future research.

Key insights

  • The EMOBENCH-UA dataset demonstrates high inter-annotator agreement (Krippendorff's alpha of 0.85), validating the robust annotation pipeline and the quality of the collected data.
  • Ukrainian-specific Transformer models generally outperform their multilingual counterparts, indicating the benefits of language-specific pre-training for this task.
  • Models trained on authentic, human-annotated Ukrainian data consistently achieve better performance than those trained on synthetic data generated by translating English corpora.
  • While cutting-edge LLMs achieved the highest overall performance, efficient encoder-based models provided highly competitive results, prompting considerations for responsible resource usage.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.