Reproduce “IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding” on OpenResearch

About this paper

Sankalp KJ, Ashutosh Kumar, Laxmaan Balaji, Nikunj Kotecha, Vinija Jain, Aman Chadha, Sreyoshi Bhaduri

A new multi-task language understanding benchmark, IndicMMLU-Pro, is introduced for nine major Indic languages, built by translating the established MMLU-Pro framework. It establishes baseline performance for state-of-the-art multilingual and Indic-specific large language models, finding GPT-4o consistently achieved the highest accuracy, ranging from 38.46% to 44.80% across languages.

Key insights

• A substantial performance gap was observed, with top general multilingual LLMs like GPT-4o significantly outperforming other models, indicating the benefits of scale and advanced architectures for Indic language understanding.
• High cosine similarity scores between the translated Indic content and the original English MMLU-Pro content confirmed strong semantic alignment and the high quality of the translated benchmark.
• The benchmark revealed varying performance patterns across different Indic languages and language families, highlighting that language-specific characteristics influence model effectiveness.
• Despite advancements in Indic-specific models, they generally did not surpass the performance of the most capable general multilingual models, suggesting ongoing challenges in specialized model development.

Summary by alphaXiv.