Reproduce “DeepSEA: A Multilingual Evaluation Benchmark for Southeast Asia” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiuzhou Han, Abhishek Bijay Mishra, Trevor Cohn, Shubham Mittal
DeepSEA evaluates language-model capabilities across 12 Southeast Asian languages using five established tasks, divided into four high-resource and eight low-resource languages. The benchmark largely uses translated source items. On CrossSum, SEA-LION-v4-27B gains 4.4 chrF points for low-resource languages and 1.0 for high-resource languages with English-pivot prompting over direct summarization; this finding is specific to that model and task.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.