Reproduce “L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rinit Jain, Tirthraj Mahajan, Advait Joshi, Raviraj Joshi
IndicQuest v2 builds a closed-book benchmark of India-specific factual questions from curricula and reference books, using language models to generate and validate candidates, semantic deduplication, human review, and translation into 19 Indic languages. It contains 3,471 English pairs across nine domains. In a ten-language evaluation, Gemma4 31B outscored Sarvam 30B in every language; this comparison does not isolate training strategy.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.