Reproduce “Through the Prism of Culture: Evaluating LLMs' Understanding of Indian Subcultures and Traditions” on OpenResearch

About this paper

Garima Chhikara, Abhishek Kumar, Abhijnan Chakraborty

The paper evaluates Large Language Models' (LLMs) comprehension of "Little Traditions" within Indian subcultures, revealing their limitations in understanding nuanced, localized cultural practices. It finds that while advanced prompting can improve performance for larger models, LLMs generally struggle with context-specific cultural understanding and perform poorly when prompted in regional Indian languages.

Key insights

• LLMs, particularly smaller models, often exhibit superficial understanding of "Little Traditions" even when selecting the correct answer, with their explanations frequently lacking cultural depth in vanilla settings.
• Automated context enrichment through LLM-generated cues substantially improves the performance of larger LLMs in culturally sensitive tasks, indicating the importance of explicit contextual information.
• Prompting LLMs in regional Indian languages generally leads to a deterioration in performance compared to English prompts, highlighting a significant language bias and the urgent need for more diverse linguistic training data.

Summary by alphaXiv.