Reproduce “Analysis of Indic Language Capabilities in LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aatman Vaidya, Tarunima Prabhakar, Denny George, Swair Shah
A review of 28 large language models and 14 evaluation studies assesses their understanding and generation capabilities across Indic languages. The research categorizes 12 frequently evaluated Indic languages into high- (e.g., Hindi, Bengali), medium-, and low-performance (e.g., Oriya, Punjabi) tiers for NLU and NLG tasks, noting performance gaps often do not directly correlate with speaker populations.
Key insights
- Only a small fraction (28 out of over 400) of publicly known LLMs demonstrate some level of usability in Indian languages, with dedicated Indic Language Models (ILLMs) consistently covering Hindi.
- MLLM performance in Indic languages is strongly influenced by their sparse representation in predominant training corpora like Common Crawl (approx. 0.47% Indic data), often leading to organizations prioritizing Hindi when supplementing data.
- Most evaluation datasets for Indic languages are direct translations of English datasets, potentially failing to capture crucial socio-cultural nuances and thus limiting comprehensive assessment.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.