Reproduce “Palm: A Culturally Inclusive and Linguistically Diverse Dataset for Arabic LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, Abdelrahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany and 36 more
PALM is a benchmark for evaluating Arabic models on country-specific cultural knowledge and dialect matching. The authors created 17,411 human-written instruction–response pairs across 22 countries, 20 topics, and ten dialects. Models show around 90% language consistency but only 2.22% to 10.56% dialect consistency; GPT-4o and Claude-3.5-Sonnet have median correctness scores above 6. The paper demonstrates evaluation, not training gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.