Reproduce “No Culture Left Behind: ArtELingo-28, a Benchmark of WikiArt with Captions in 28 Languages” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Youssef Mohamed, Runjia Li, Ibrahim Said Ahmad, Kilichbek Haydarov, Philip Torr, Kenneth Ward Church, Mohamed Elhoseiny

ArtELingo-28 introduces a benchmark dataset for affective vision-language understanding, providing 200,000 new emotion labels and subjective textual explanations for 2000 WikiArt images across 28 languages, including 25 low-resource ones. This resource facilitates the evaluation of multimodal AI models in generating culturally-nuanced emotional explanations and reveals cross-cultural patterns in artistic perception.

Key insights

  • Kullback-Leibler divergence analysis on emotion distributions revealed distinct language groupings (e.g., African and Asian clusters), suggesting cultural influences on emotion perception and labeling.
  • Few-shot fine-tuning significantly improved performance for low-resource languages, but increasing data beyond a certain point yielded diminishing returns, implying horizontal dataset expansion (more languages) may be more impactful than vertical (more samples per language).
  • Language clustering based on ROUGE scores in the One-vs-All Zero-Shot setup aligned with real-world cultural connections rather than just shared writing systems, indicating the models captured underlying cultural signals.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.