Reproduce “UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languages” on OpenResearch

About this paper

Pranjal A. Chitale, Varun Gumma, Sanchit Ahuja, Prashant Kodali, Manan Uppadhyay, Deepthi Sudharsan, Sunayana Sitaram

Researchers developed UPDESH, a 9.5 million data point synthetic instruction-following dataset for 13 Indic languages and English, generated through a bottom-up approach grounded in language-specific Wikipedia content. Models fine-tuned on UPDESH consistently achieve superior performance in NLU and NLG tasks and demonstrate enhanced cultural relevance compared to existing multilingual datasets.

Key insights

• Bottom-up synthetic data generation, when grounded in language-specific content, is effective in creating high-quality, culturally aware, and linguistically natural training data for low-resource languages.
• The combination of translated reasoning data and culturally grounded generative data provides complementary benefits, leading to a more robust and superior overall multilingual model performance compared to purely translated or single-type datasets.
• Current LLM-as-judge paradigms have limitations in reliably assessing culturally and linguistically nuanced aspects (e.g., linguistic plausibility, persona adherence), highlighting the necessity for calibrated human evaluation in such contexts.

Summary by alphaXiv.