Reproduce “SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V. Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov and 53 more
SEACrowd establishes a comprehensive, centralized data hub and benchmark suite for Southeast Asian languages, consolidating nearly 500 corpora across text, image, and audio for approximately 1,000 languages. This initiative reveals that state-of-the-art large language models often generate "translationese" rather than natural text for many SEA languages, performing naturally less than 20% of the time for certain indigenous languages.
Key insights
- Multilingual large language models (LLMs) like AYA-101 and mT0 generally exhibit superior zero-shot performance and greater language equity across SEA languages compared to English-only or country-specific models.
- LLMs often produce
- translationese
- rather than natural language for many SEA languages, with some indigenous languages showing naturalness rates below 20%, highlighting a significant quality and cultural authenticity gap.
- A vast resource gap persists for SEA languages, with most languages having limited datasets, particularly in speech and vision-language modalities, and a majority of existing data lacking cultural relevance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.