Reproduce “CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks” on OpenResearch

About this paper

Peiqin Lin, Chenyang Lyu, Wenjiang Luo, Haotian Ye, Md Mehrab Hossain, Chunlan Ma, Shaoxiong Ji, Younes Samih and 17 more

The CulturALL benchmark provides a comprehensive evaluation of Large Language Models' multilingual and multicultural competence on grounded, real-world tasks, revealing that even state-of-the-art models achieved a maximum accuracy of 44.48%. This indicates substantial room for improvement in LLM understanding of complex, culturally nuanced scenarios.

Key insights

• The human-LLM collaborative framework offers a scalable and effective method for creating high-quality, diverse, and challenging datasets by leveraging both expert human insight and LLM efficiency.
• Grounded tasks inherently necessitate a complex integration of linguistic interpretation, precise cultural knowledge, and contextual understanding, making them distinct from simpler knowledge recall tasks.
• Evaluating LLMs using native language prompts consistently yields higher accuracy (8.08% improvement for one model) compared to English translations, suggesting that translation can obscure vital cultural nuances.

Summary by alphaXiv.